Image coding method on basis of non-separable secondary transform and device therefor

ABSTRACT

An image decoding method performed by means of a decoding device according to the present disclosure comprises the steps of: deriving transform coefficients of a target block from a bitstream; deriving a non-separable secondary transform (NSST) index with respect to the target block; performing inverse transform with respect to the transform coefficients of the target block on the basis of the NSST index and thus deriving residual samples of the target block; and generating a reconstructed picture on the basis of the residual samples.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/316,049, filed on May 10, 2021, which is a continuation of U.S.application Ser. No. 16/901,857, filed on Jun. 15, 2020, now U.S. Pat.No. 11,039,140, which is a continuation pursuant to 35 U.S.C. § 119(e)of International Application PCT/KR2018/015812, with an internationalfiling date of Dec. 13, 2018, which claims the benefit of U.S.Provisional Application No. 62/599,021 filed on Dec. 15, 2017, thecontents of which are all hereby incorporated by reference herein intheir entirety.

BACKGROUND OF THE DISCLOSURE Field of the Disclosure

The present disclosure relates to a video coding technique, and moreparticularly, to a video decoding method and apparatus thereforaccording to non-separable secondary transform in a video coding system.

Related Art

Demand for high-resolution, high-quality images such High Definition(HD) images and Ultra High Definition (UHD) images have been increasingin various fields. As the image data has high resolution and highquality, the amount of information or bits to be transmitted increasesrelative to the legacy image data. Therefore, when image data istransmitted using a medium such as a conventional wired/wirelessbroadband line or image data is stored using an existing storage medium,the transmission cost and the storage cost thereof are increased.

Accordingly, there is a need for a highly efficient image compressiontechnique for effectively transmitting, storing, and reproducinginformation of high resolution and high quality images.

SUMMARY OF THE DISCLOSURE

The present disclosure provides a method and apparatus for improvingvideo coding efficiency.

The present disclosure also provides a video decoding method andapparatus for applying NSST for a target block.

The present disclosure also provides a video decoding method andapparatus for deriving a range of NSST index based on a specificcondition of a target block.

The present disclosure also provides a video decoding method andapparatus for determining whether to code NSST index based on atransform coefficient of a target block.

According to an embodiment of the present disclosure, it is provided avideo decoding method performed by a decoding apparatus. The methodincludes deriving transform coefficients of a target block from abitstream; deriving a Non-Separable Secondary Transform (NSST) index forthe target block; deriving residual samples of the target block byperforming inverse transforming for the transform coefficients of thetarget block based on the NSST index; and generating a reconstructedpicture based on the residual samples.

According to another embodiment of the present disclosure, it isprovided a decoding apparatus performing a video decoding. The decodingapparatus includes an entropy decoder for deriving transformcoefficients of a target block from a bitstream and deriving aNon-Separable Secondary Transform (NSST) index for the target block; aninverse transformer for deriving residual samples of the target block byperforming inverse transforming for the transform coefficients of thetarget block based on the NSST index; and an adder for generating areconstructed picture based on the residual samples.

According to still another embodiment of the present disclosure, it isprovided a video encoding method performed by an encoding apparatus. Themethod includes deriving residual samples of a target block; derivingtransform coefficients of the target block by performing transform forthe residual samples; determining whether to encode a Non-SeparableSecondary Transform (NSST) index for the transform coefficients; andencoding information for transform coefficients, wherein the step ofdetermining whether to encode the NSST index includes: scanning R+1th toNth transform coefficients among the transform coefficients of thetarget block; and when a non-zero transform coefficient is included inthe R+1th to Nth transform coefficients, determining not to encode theNSST index, and wherein the N is a sample number of the top-left targetarea, and the R is a reduced coefficient, and wherein the R is smallerthan the N.

According to still another embodiment of the present disclosure, it isprovided a video encoding apparatus. The encoding apparatus includes anadder for deriving residual samples of a target block; a transformer forderiving transform coefficients of the target block by performingtransform for the residual samples; determining whether to encode aNon-Separable Secondary Transform (NSST) index for the transformcoefficients; and an entropy encoder for encoding information fortransform coefficients, wherein the entropy encoder scans R+1th to Nthtransform coefficients among the transform coefficients of the targetblock; and when a non-zero transform coefficient is included in theR+1th to Nth transform coefficients, determines not to encode the NSSTindex, and wherein the N is a sample number of the top-left target area,and the R is a reduced coefficient, and wherein the R is smaller thanthe N.

According to the present disclosure, a range of NSST index may bederived based on a specific condition of a target block, and throughthis, bit amount for the NSST index may be reduced, and accordingly,overall coding rate may be improved.

According to the present disclosure, a transmission of syntax elementfor NSST index may be determined based on transform coefficients for atarget block, and through this, bit amount for the NSST index may bereduced, and accordingly, overall coding rate may be improved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a configuration of a videoencoding apparatus to which the present disclosure is applicable.

FIG. 2 illustrates an example of a video encoding method performed by avideo encoding apparatus.

FIG. 3 is a schematic diagram illustrating a configuration of a videodecoding apparatus to which the present disclosure is applicable.

FIG. 4 illustrates an example of a video decoding method performed by adecoding apparatus.

FIG. 5 schematically illustrates a multi-transform scheme according tothe present disclosure.

FIG. 6 illustrates 65 intra direction modes of a prediction direction.

FIG. 7a and FIG. 7b are flowcharts illustrating a coding process of atransform coefficient according to an embodiment.

FIG. 8 is a diagram for describing an array of transform coefficientsbased on a target block according to an embodiment of the presentdisclosure.

FIG. 9 illustrates an example of scanning transform coefficients fromR+1 to N.

FIG. 10a and FIG. 10b are flowcharts illustrating a coding process of anNSST index according to an embodiment.

FIG. 11 illustrates an example of determining whether an NSST index iscoded.

FIG. 12 illustrates an example of scanning transform coefficients fromR+1 to N for all components of a target block.

FIG. 13 schematically illustrates a video encoding method by an encodingapparatus according to the present disclosure.

FIG. 14 schematically illustrates an encoding apparatus performing avideo encoding method according to the present disclosure.

FIG. 15 schematically illustrates a video decoding method by a decodingapparatus according to the present disclosure.

FIG. 16 schematically illustrates the decoding apparatus performing thevideo decoding method according to the present disclosure.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

The present disclosure may be modified in various forms, and specificembodiments thereof will be described and illustrated in the drawings.However, the embodiments are not intended for limiting the disclosure.The terms used in the following description are used to merely describespecific embodiments, but are not intended to limit the disclosure. Anexpression of a singular number includes an expression of the pluralnumber, so long as it is clearly read differently. The terms such as“include” and “have” are intended to indicate that features, numbers,steps, operations, elements, components, or combinations thereof used inthe following description exist and it should be thus understood thatthe possibility of existence or addition of one or more differentfeatures, numbers, steps, operations, elements, components, orcombinations thereof is not excluded.

Meanwhile, elements in the drawings described in the disclosure areindependently drawn for the purpose of convenience for explanation ofdifferent specific functions, and do not mean that the elements areembodied by independent hardware or independent software. For example,two or more elements of the elements may be combined to form a singleelement, or one element may be divided into plural elements. Theembodiments in which the elements are combined and/or divided belong tothe disclosure without departing from the concept of the disclosure.

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the accompanying drawings. In addition, likereference numerals are used to indicate like elements throughout thedrawings, and the same descriptions on the like elements will beomitted.

Meanwhile, the present disclosure relates to video/image coding. Forexample, the method(s)/embodiment(s) disclosed in the present disclosuremay be applied to a method disclosed in a versatile video coding (VVC)standard or a next generation video/image coding standard.

In the present specification, generally a picture means a unitrepresenting an image at a specific time, a slice is a unit constitutinga part of the picture. One picture may be composed of plural slices, andthe terms of a picture and a slice may be mixed with each other asoccasion demands.

A pixel or a pel may mean a minimum unit constituting one picture (orimage). Further, a “sample” may be used as a term corresponding to apixel. The sample may generally represent a pixel or a value of a pixel,may represent only a pixel (a pixel value) of a luma component, and mayrepresent only a pixel (a pixel value) of a chroma component.

A unit indicates a basic unit of image processing. The unit may includeat least one of a specific area and information related to the area.Optionally, the unit may be mixed with terms such as a block, an area,or the like. In a typical case, an M×N block may represent a set ofsamples or transform coefficients arranged in M columns and N rows.

FIG. 1 is a schematic diagram illustrating a configuration of a videoencoding device to which the present disclosure is applicable.

Referring to FIG. 1, a video encoding device (100) may include a picturepartitioner (105), a predictor (110), a residual processor (120), anentropy encoder (130), an adder (140), a filter (150), and a memory(160). The residual processor (120) may include a subtractor (121), atransformer (122), a quantizer (123), a re-arranger (124), a dequantizer(125), an inverse transformer (126).

The picture partitioner (105) may split an input picture into at leastone processing unit.

In an example, the processing unit may be referred to as a coding unit(CU). In this case, the coding unit may be recursively split from thelargest coding unit (LCU) according to a quad-tree binary-tree (QTBT)structure. For example, one coding unit may be split into a plurality ofcoding units of a deeper depth based on a quadtree structure and/or abinary tree structure. In this case, for example, the quad treestructure may be first applied and the binary tree structure may beapplied later. Alternatively, the binary tree structure may be appliedfirst. The coding procedure according to the present disclosure may beperformed based on a final coding unit which is not split any further.In this case, the largest coding unit may be used as the final codingunit based on coding efficiency, or the like, depending on imagecharacteristics, or the coding unit may be recursively split into codingunits of a lower depth as necessary and a coding unit having an optimalsize may be used as the final coding unit. Here, the coding proceduremay include a procedure such as prediction, transformation, andreconstruction, which will be described later.

In another example, the processing unit may include a coding unit (CU)prediction unit (PU), or a transform unit (TU). The coding unit may besplit from the largest coding unit (LCU) into coding units of a deeperdepth according to the quad tree structure. In this case, the largestcoding unit may be directly used as the final coding unit based on thecoding efficiency, or the like, depending on the image characteristics,or the coding unit may be recursively split into coding units of adeeper depth as necessary and a coding unit having an optimal size maybe used as a final coding unit. When the smallest coding unit (SCU) isset, the coding unit may not be split into coding units smaller than thesmallest coding unit. Here, the final coding unit refers to a codingunit which is partitioned or split to a prediction unit or a transformunit. The prediction unit is a unit which is partitioned from a codingunit, and may be a unit of sample prediction. Here, the prediction unitmay be divided into sub-blocks. The transform unit may be divided fromthe coding unit according to the quad-tree structure and may be a unitfor deriving a transform coefficient and/or a unit for deriving aresidual signal from the transform coefficient. Hereinafter, the codingunit may be referred to as a coding block (CB), the prediction unit maybe referred to as a prediction block (PB), and the transform unit may bereferred to as a transform block (TB). The prediction block orprediction unit may refer to a specific area in the form of a block in apicture and include an array of prediction samples. Also, the transformblock or transform unit may refer to a specific area in the form of ablock in a picture and include the transform coefficient or an array ofresidual samples.

The predictor (110) may perform prediction on a processing target block(hereinafter, a current block), and may generate a predicted blockincluding prediction samples for the current block. A unit of predictionperformed in the predictor (110) may be a coding block, or may be atransform block, or may be a prediction block.

The predictor (110) may determine whether intra-prediction is applied orinter-prediction is applied to the current block. For example, thepredictor (110) may determine whether the intra-prediction or theinter-prediction is applied in unit of CU.

In case of the intra-prediction, the predictor (110) may derive aprediction sample for the current block based on a reference sampleoutside the current block in a picture to which the current blockbelongs (hereinafter, a current picture). In this case, the predictor(110) may derive the prediction sample based on an average orinterpolation of neighboring reference samples of the current block(case (i)), or may derive the prediction sample based on a referencesample existing in a specific (prediction) direction as to a predictionsample among the neighboring reference samples of the current block(case (ii)). The case (i) may be called a non-directional mode or anon-angular mode, and the case (ii) may be called a directional mode oran angular mode. In the intra-prediction, prediction modes may includeas an example 33 directional modes and at least two non-directionalmodes. The non-directional modes may include DC mode and planar mode.The predictor (110) may determine the prediction mode to be applied tothe current block by using the prediction mode applied to theneighboring block.

In case of the inter-prediction, the predictor (110) may derive theprediction sample for the current block based on a sample specified by amotion vector on a reference picture. The predictor (110) may derive theprediction sample for the current block by applying any one of a skipmode, a merge mode, and a motion vector prediction (MVP) mode. In caseof the skip mode and the merge mode, the predictor (110) may use motioninformation of the neighboring block as motion information of thecurrent block. In case of the skip mode, unlike in the merge mode, adifference (residual) between the prediction sample and an originalsample is not transmitted. In case of the MVP mode, a motion vector ofthe neighboring block is used as a motion vector predictor and thus isused as a motion vector predictor of the current block to derive amotion vector of the current block.

In case of the inter-prediction, the neighboring block may include aspatial neighboring block existing in the current picture and a temporalneighboring block existing in the reference picture. The referencepicture including the temporal neighboring block may also be called acollocated picture (colPic). Motion information may include the motionvector and a reference picture index. Information such as predictionmode information and motion information may be (entropy) encoded, andthen output as a form of a bitstream.

When motion information of a temporal neighboring block is used in theskip mode and the merge mode, a highest picture in a reference picturelist may be used as a reference picture. Reference pictures included inthe reference picture list may be aligned based on a picture order count(POC) difference between a current picture and a corresponding referencepicture. A POC corresponds to a display order and can be discriminatedfrom a coding order.

The subtractor (121) generates a residual sample which is a differencebetween an original sample and a prediction sample. If the skip mode isapplied, the residual sample may not be generated as described above.

The transformer (122) transforms residual samples in units of atransform block to generate a transform coefficient. The transformer(122) may perform transformation based on the size of a correspondingtransform block and a prediction mode applied to a coding block orprediction block spatially overlapping with the transform block. Forexample, residual samples can be transformed using discrete sinetransform (DST) transform kernel if intra-prediction is applied to thecoding block or the prediction block overlapping with the transformblock and the transform block is a 4×4 residual array and is transformedusing discrete cosine transform (DCT) transform kernel in other cases.

The quantizer (123) may quantize the transform coefficients to generatequantized transform coefficients.

The re-arranger (124) rearranges quantized transform coefficients. There-arranger (124) may rearrange the quantized transform coefficients inthe form of a block into a one-dimensional vector through a coefficientscanning method. Although the re-arranger (124) is described as aseparate component, the re-arranger (124) may be a part of the quantizer(123).

The entropy encoder (130) may perform entropy-encoding on the quantizedtransform coefficients. The entropy encoding may include an encodingmethod, for example, an exponential Golomb, a context-adaptive variablelength coding (CAVLC), a context-adaptive binary arithmetic coding(CABAC), or the like. The entropy encoder (130) may perform encodingtogether or separately on information (e.g., a syntax element value orthe like) required for video reconstruction in addition to the quantizedtransform coefficients. The entropy-encoded information may betransmitted or stored in unit of a network abstraction layer (NAL) in abitstream form.

The dequantizer (125) dequantizes values (transform coefficients)quantized by the quantizer (123) and the inverse transformer (126)inversely transforms values dequantized by the dequantizer (125) togenerate a residual sample.

The adder (140) adds a residual sample to a prediction sample toreconstruct a picture. The residual sample may be added to theprediction sample in units of a block to generate a reconstructed block.Although the adder (140) is described as a separate component, the adder(140) may be a part of the predictor (110). Meanwhile, the adder (140)may be referred to as a reconstructor or reconstructed block generator.

The filter (150) may apply deblocking filtering and/or a sample adaptiveoffset to the reconstructed picture. Artifacts at a block boundary inthe reconstructed picture or distortion in quantization can be correctedthrough deblocking filtering and/or sample adaptive offset. Sampleadaptive offset may be applied in units of a sample after deblockingfiltering is completed. The filter (150) may apply an adaptive loopfilter (ALF) to the reconstructed picture. The ALF may be applied to thereconstructed picture to which deblocking filtering and/or sampleadaptive offset has been applied.

The memory (160) may store a reconstructed picture (decoded picture) orinformation necessary for encoding/decoding. Here, the reconstructedpicture may be the reconstructed picture filtered by the filter (150).The stored reconstructed picture may be used as a reference picture for(inter) prediction of other pictures. For example, the memory (160) maystore (reference) pictures used for inter-prediction. Here, picturesused for inter-prediction may be designated according to a referencepicture set or a reference picture list.

FIG. 2 illustrates an example of a video encoding method performed by avideo encoding apparatus. Referring to FIG. 2, the video encoding methodmay include procedures of intra/inter prediction, transform,quantization and entropy encoding. For example, a prediction block of acurrent block may be generated through the intra/inter prediction and aresidual block of the current block may be generated through subtractionbetween an input block of the current block and the prediction block.Later, through a transform for the residual block, a coefficient block,that is, transform coefficients for the current block may be generated.The transform coefficients may be quantized and entropy-encoded andthen, stored in a bitstream.

FIG. 3 is a schematic diagram illustrating a configuration of a videodecoding apparatus to which the present disclosure is applicable.

Referring to FIG. 3, a video decoding apparatus 300 includes an entropydecoder 310, a residual processor 320, a predictor 330, an adder 340, afilter 350 and a memory 360. Here, the residual processor 320 mayinclude a re-arranger 321, a dequantizer 322 and an inverse transformer323.

When a bitstream including video information is input, the videodecoding apparatus 300 may reconstruct a video corresponding to aprocess in which the video information is processed in the videoencoding apparatus.

For example, the video decoding apparatus 300 may perform video decodingusing a processor applied in the video encoding apparatus. Thus, theprocessor of video decoding may be a coding unit, for example, or may bea coding unit, a prediction unit or a transform unit, for anotherexample. The coding unit may be partitioned according to a quad treestructure and/or binary tree structure from the largest coding unit.

A prediction unit and a transform unit may be further used in somecases, and in this case, the prediction block is a block derived orpartitioned from the coding unit and may be a unit of sample prediction.Here, the prediction unit may be divided into sub-blocks. The transformunit may be split from the coding unit according to the quad treestructure and may be a unit that derives a transform coefficient or aunit that derives a residual signal from the transform coefficient.

The entropy decoder 310 may parse the bitstream to output informationrequired for video reconstruction or picture reconstruction. Forexample, the entropy decoder 310 may decode information in the bitstreambased on a coding method such as exponential Golomb encoding, CAVLC,CABAC, or the like, and may output a value of a syntax element requiredfor video reconstruction and a quantized value of a transformcoefficient regarding a residual.

More specifically, a CABAC entropy decoding method may receive a bincorresponding to each syntax element in a bitstream, determine a contextmodel using decoding target syntax element information and decodinginformation of neighboring and decoding target blocks or information ofa symbol/bin decoded in a previous step, predict bin generationprobability according to the determined context model and performarithmetic decoding of the bin to generate a symbol corresponding toeach syntax element value. Here, the CABAC entropy decoding method mayupdate the context model using information of a symbol/bin decoded for acontext model of the next symbol/bin after determination of the contextmodel.

Information on prediction among information decoded in the entropydecoder 310 may be provided to the predictor 350 and residual values,that is, quantized transform coefficients, on which entropy decoding hasbeen performed by the entropy decoder 310 may be input to there-arranger 321.

The re-arranger 321 may rearrange the quantized transform coefficientsinto a two-dimensional block form. The re-arranger 321 may performrearrangement corresponding to coefficient scanning performed by theencoding apparatus. Although the re-arranger 321 is described as aseparate component, the re-arranger 321 may be a part of the dequantizer322.

The dequantizer 322 may de-quantize the quantized transform coefficientsbased on a (de)quantization parameter to output a transform coefficient.In this case, information for deriving a quantization parameter may besignaled from the encoding apparatus.

The inverse transformer 323 may inverse-transform the transformcoefficients to derive residual samples.

The predictor 330 may perform prediction on a current block and maygenerate a predicted block including prediction samples for the currentblock. A unit of prediction performed in the predictor 330 may be acoding block or may be a transform block or may be a prediction block.

The predictor 330 may determine whether to apply intra-prediction orinter-prediction based on information on a prediction. In this case, aunit for determining which one will be used between the intra-predictionand the inter-prediction may be different from a unit for generating aprediction sample. In addition, a unit for generating the predictionsample may also be different in the inter-prediction and theintra-prediction. For example, which one will be applied between theinter-prediction and the intra-prediction may be determined in unit ofCU. Further, for example, in the inter-prediction, the prediction samplemay be generated by determining the prediction mode in unit of PU, andin the intra-prediction, the prediction sample may be generated in unitof TU by determining the prediction mode in unit of PU.

In case of the intra-prediction, the predictor 330 may derive aprediction sample for a current block based on a neighboring referencesample in a current picture. The predictor 330 may derive the predictionsample for the current block by applying a directional mode or anon-directional mode based on the neighboring reference sample of thecurrent block. In this case, a prediction mode to be applied to thecurrent block may be determined by using an intra-prediction mode of aneighboring block.

In the case of inter-prediction, the predictor 330 may derive aprediction sample for a current block based on a sample specified in areference picture according to a motion vector. The predictor 330 mayderive the prediction sample for the current block using one of the skipmode, the merge mode and the MVP mode. Here, motion information requiredfor inter-prediction of the current block provided by the video encodingapparatus, for example, a motion vector and information on a referencepicture index may be acquired or derived based on the information onprediction.

In the skip mode and the merge mode, motion information of a neighboringblock may be used as motion information of the current block. Here, theneighboring block may include a spatial neighboring block and a temporalneighboring block.

The predictor 330 may construct a merge candidate list using motioninformation of available neighboring blocks and use informationindicated by a merge index on the merge candidate list as a motionvector of the current block. The merge index may be signaled by theencoding apparatus. Motion information may include a motion vector and areference picture. When motion information of a temporal neighboringblock is used in the skip mode and the merge mode, a highest picture ina reference picture list may be used as a reference picture.

In the case of the skip mode, a difference (residual) between aprediction sample and an original sample is not transmitted,distinguished from the merge mode.

In the case of the MVP mode, the motion vector of the current block maybe derived using a motion vector of a neighboring block as a motionvector predictor. Here, the neighboring block may include a spatialneighboring block and a temporal neighboring block.

When the merge mode is applied, for example, a merge candidate list maybe generated using a motion vector of a reconstructed spatialneighboring block and/or a motion vector corresponding to a Col blockwhich is a temporal neighboring block. A motion vector of a candidateblock selected from the merge candidate list is used as the motionvector of the current block in the merge mode. The aforementionedinformation on prediction may include a merge index indicating acandidate block having the best motion vector selected from candidateblocks included in the merge candidate list. Here, the predictor 330 mayderive the motion vector of the current block using the merge index.

When the MVP (Motion vector Prediction) mode is applied as anotherexample, a motion vector predictor candidate list may be generated usinga motion vector of a reconstructed spatial neighboring block and/or amotion vector corresponding to a Col block which is a temporalneighboring block. That is, the motion vector of the reconstructedspatial neighboring block and/or the motion vector corresponding to theCol block which is the temporal neighboring block may be used as motionvector candidates. The aforementioned information on prediction mayinclude a prediction motion vector index indicating the best motionvector selected from motion vector candidates included in the list.Here, the predictor 330 may select a prediction motion vector of thecurrent block from the motion vector candidates included in the motionvector candidate list using the motion vector index. The predictor ofthe encoding apparatus may obtain a motion vector difference (MVD)between the motion vector of the current block and a motion vectorpredictor, encode the MVD and output the encoded MVD in the form of abitstream. That is, the MVD may be obtained by subtracting the motionvector predictor from the motion vector of the current block. Here, thepredictor 330 may acquire a motion vector included in the information onprediction and derive the motion vector of the current block by addingthe motion vector difference to the motion vector predictor. Inaddition, the predictor may obtain or derive a reference picture indexindicating a reference picture from the aforementioned information onprediction.

The adder 340 may add a residual sample to a prediction sample toreconstruct a current block or a current picture. The adder 340 mayreconstruct the current picture by adding the residual sample to theprediction sample in units of a block. When the skip mode is applied, aresidual is not transmitted and thus the prediction sample may become areconstructed sample. Although the adder 340 is described as a separatecomponent, the adder 340 may be a part of the predictor 330. Meanwhile,the adder 340 may be referred to as a reconstructor or reconstructedblock generator.

The filter 350 may apply deblocking filtering, sample adaptive offsetand/or ALF to the reconstructed picture. Here, sample adaptive offsetmay be applied in units of a sample after deblocking filtering. The ALFmay be applied after deblocking filtering and/or application of sampleadaptive offset.

The memory 360 may store a reconstructed picture (decoded picture) orinformation necessary for decoding. Here, the reconstructed picture maybe the reconstructed picture filtered by the filter 350. For example,the memory 360 may store pictures used for inter-prediction. Here, thepictures used for inter-prediction may be designated according to areference picture set or a reference picture list. A reconstructedpicture may be used as a reference picture for other pictures. Thememory 360 may output reconstructed pictures in an output order.

FIG. 4 illustrates an example of a video decoding method performed by adecoding apparatus. Referring to FIG. 4, the video decoding method mayinclude procedures of entropy decoding, dequantization, inversetransform and intra/inter prediction. For example, inverse procedures ofthe encoding method may be performed in the decoding apparatus.Particularly, through entropy decoding for a bitstream, quantizedtransform coefficients may be obtained and through dequantizationprocedure for the quantized transform coefficients, a coefficient blockof the current block, that is, transform coefficients may be obtained.Through inverse transform for the transform coefficients, a residualblock of the current block may be derived, and through summation of aprediction block of the current block and the residual block derivedthrough intra/inter prediction, a reconstructed block of the currentblock may be derived.

Meanwhile, through the transform described above, transform coefficientsof lower frequency for the residual block of the current block may bederived, and at an end of the residual block, a zero tail may bederived.

Particularly, the transform may include two types of main procedures,and the main procedures may include a core transform and a secondarytransform. The transform including the core transform and the secondarytransform may be represented as multi-transform scheme.

FIG. 5 schematically illustrates a multi-transform scheme according tothe present disclosure.

Referring to FIG. 5, the transformer may correspond to the transformerof the encoding apparatus of FIG. 1, and the inverse transformer maycorrespond to the inverse transformer of the encoding apparatus of FIG.1 or the inverse transformer of the decoding apparatus of FIG. 3.

The transformer may derive (primary) transform coefficients byperforming a primary transform based on residual samples (residualsample array) within a residual block (S510). In this case, the primarytransform may include an Adaptive Multiple core Transform (AMT). Theadaptive multiple core transform may also be represented as a MultipleTransform Set (MTS).

The adaptive multiple core transform may indicate a method of performinga transform additionally using a discrete cosine transform (DCT) Type 2,a discrete sine transform (DST) Type 7, a DCT Type 8 and/or a DSTType 1. That is, the adaptive multiple core transform may indicate atransform method of transforming a residual signal (or residual block)of a spatial domain into transform coefficients (or primary transformcoefficients) of a frequency domain based on a plurality of transformkernels selected among the DCT Type 2, the DST Type 7, the DCT Type 8and the DST Type 1. In this case, the primary transform coefficients maybe called temporary transform coefficients from the viewpoint of thetransformer.

In other words, in the case that the existing transform method isapplied, transform coefficients may be generated by applying a transformfrom a spatial domain for a residual signal (or residual block) to afrequency domain based on the DCT Type 2. On the other hand, in the casethat the adaptive multi-core transform is applied, transformcoefficients (or primary transform coefficients) may be generated byapplying a transform from a spatial domain for a residual signal (orresidual block) to a frequency domain based on the DCT Type 2, the DSTType 7, the DCT Type 8 and/or the DST Type 1. In this case, the DCT Type2, the DST Type 7, the DCT Type 8 and the DST Type 1 may be called atransform type, a transform kernel or a transform core.

For reference, the DCT/DST transform types may be defined based on basisfunctions. The basis functions may be represented as follows.

TABLE 1 Transform Type Basis function T_(i)(j), i, j = 0,1, . . . , N-1DCT-II${T_{i}(j)} = {\omega_{0} \cdot \sqrt{\frac{2}{N}} \cdot {\cos\left( \frac{\pi \cdot i \cdot \left( {{2j} + 1} \right)}{2N} \right)}}$${{where}\omega_{0}} = \left\{ \begin{matrix}\sqrt{\frac{2}{N}} & {i = 0} \\1 & {i \neq 0}\end{matrix} \right.$ DCT-V${{T_{i}(j)} = {\omega_{0} \cdot \omega_{1} \cdot \sqrt{\frac{2}{{2N} - 1}} \cdot {\cos\left( \frac{2{\pi \cdot i \cdot j}}{{2N} - 1} \right)}}},$${{where}\omega_{0}} = \left\{ {\begin{matrix}\sqrt{\frac{2}{N}} & {i = 0} \\1 & {i \neq 0}\end{matrix},} \right.$ $\omega_{1}\left\{ \begin{matrix}\sqrt{\frac{2}{N}} & {j = 0} \\1 & {j \neq 0}\end{matrix} \right.$ DCT-VIII${T_{i}(j)} = {\sqrt{\frac{4}{{2N} + 1}} \cdot {\cos\left( \frac{\pi \cdot \left( {{2i} + 1} \right) \cdot \left( {{2j} + 1} \right)}{{4N} + 2} \right)}}$DST-I${T_{i}(j)} = {\sqrt{\frac{4}{N + 1}} \cdot {\sin\left( \frac{\pi \cdot \left( {i + 1} \right) \cdot \left( {j + 1} \right)}{N + 1} \right)}}$DST-VII${T_{i}(j)} = {\sqrt{\frac{4}{{2N} + 1}} \cdot {\sin\left( \frac{\pi \cdot \left( {{2i} + 1} \right) \cdot \left( {j + 1} \right)}{{2N} + 1} \right)}}$

In the case that the adaptive multi-core transform is performed, avertical transform kernel and a horizontal transform kernel for a targetblock may be selected among transform kernels. A vertical transform fora target block may be performed based on the vertical transform kernel.A horizontal transform for the target block may be performed based onthe horizontal transform kernel. In this case, the horizontal transformmay represent a transform for the horizontal components of the targetblock. The vertical transform may represent a transform for the verticalcomponents of the target block. The vertical transform kernel/horizontaltransform kernel may be adaptively determined based on a prediction modeof the target block (CU or sub-block) encompassing a residual blockand/or a transform index indicative of a transform subset.

For example, the adaptive multi-core transform may be applied to thecase that both of a width and a height of a target block are equal to orsmaller than 64, and it may be determined based on a CU level flagwhether the adaptive multi-core transform is applied. Particularly, inthe case that the CU level flag is 0, the existing transform methoddescribed above may be applied. That is, in the case that the CU levelflag is 0, the transform from a spatial domain for a residual signal (orresidual block) to a frequency domain based on the DCT Type 2 may beapplied and transform coefficients may be generated, and the transformcoefficients may be encoded. Meanwhile, here, the target block may be aCU. In the case that the CU level flag is 0, the adaptive multi-coretransform may be applied to the target block.

In addition, in the case of a luma block of the target block to whichthe adaptive multi-core transform is applied, two additional flags maybe signaled, and based on the flags, a vertical transform kernel and ahorizontal transform kernel may be selected. The flag for the verticaltransform kernel may be represented as AMT vertical flag, andAMT_TU_vertical flag (or EMT_TU_vertical flag) may represent a syntaxelement of the AMT vertical flag. The flag for the horizontal transformkernel may be represented as AMT horizontal flag, andAMT_TU_horizontal_flag (or EMT_TU_horizontal_flag) may represent asyntax element of the AMT horizontal flag. The AMT vertical flag mayindicate one transform kernel candidate among the transform kernelcandidates included in a transform subset for the vertical transformkernel, and the transform kernel candidate indicated by the AMT verticalflag may be derived as a vertical transform kernel for the target block.Furthermore, the AMT horizontal flag may indicate one transform kernelcandidate among the transform kernel candidates included in a transformsubset for the horizontal transform kernel, and the transform kernelcandidate indicated by the AMT horizontal flag may be derived as ahorizontal transform kernel for the target block. Meanwhile, the AMTvertical flag may be represented as MTS vertical flag, and the AMThorizontal flag may be represented as MTS horizontal flag.

Meanwhile, three transform subsets may be preconfigured, and based onthe intra prediction mode applied to the target block, one of thetransform subsets may be derived as a transform subset for the verticaltransform kernel. In addition, one of the transform subsets may bederived as a transform subset for the horizontal transform kernel basedon the intra prediction mode applied to the target block. For example,the preconfigured transform subsets may be derived as represented in thefollowing Table.

TABLE 2 Transform Set Transform Candidates 0 DST-VII, DCT-VIII 1DST-VII, DST-I 2 DST-VII, DCT-VIII

Referring to Table 2, the transform subset of which index value is 0 mayrepresent a transform subset including DST type 7 and DCT type 7 as atransform kernel candidate. The transform subset of which index value is1 may represent a transform subset including DST type 7 and DCT type 1as a transform kernel candidate. The transform subset of which indexvalue is 2 may represent a transform subset including DST type 7 and DCTtype 8 as a transform kernel candidate.

The transform subset for the vertical transform kernel and the transformsubset for the horizontal transform kernel derived based on the intraprediction mode applied to the target block may be derived asrepresented in the following Table.

TABLE 3 Intra Mode 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 V 2 1 0 10 1 0 1 0 1 0 1 0 1 0 0 0 0 H 2 1 0 1 0 1 0 1 0 1 0 1 0 1 2 2 2 2 IntraMode 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 V 0 0 0 0 0 1 01 0 1 0 1 0 1 0 1 0 H 2 2 2 2 2 1 0 1 0 1 0 1 0 1 0 1 0 Intra Mode 35 3637 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 V 1 0 1 0 1 0 1 0 1 0 12 2 2 2 2 2 2 H 1 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 Intra Mode 53 54 5556 57 58 59 60 61 62 63 64 65 66 V 2 2 1 0 1 0 1 0 1 0 1 0 1 0 H 0 0 1 01 0 1 0 1 0 1 0 1 0

Herein, V represents the transform subset for the vertical transformkernel and P represents the transform subset for the horizontaltransform kernel.

In the case that a value of the AMVT flag (or EMT_CU_flag) is 1, asrepresented in Table 3, the transform subset for the vertical transformkernel and the transform subset for the horizontal transform kernel maybe derived based on the intra prediction mode of the target block.Later, among the transform kernel candidates included in the transformsubset for the vertical transform kernel, a transform kernel candidateindicated by the AMT vertical flag of the target block may be derived asthe vertical transform kernel of the target block, and among thetransform kernel candidates included in the transform subset for thehorizontal transform kernel, a transform kernel candidate indicated bythe AMT horizontal flag of the target block may be derived as thehorizontal transform kernel of the target block. Meanwhile, the AMT flagmay also be represented as MTS flag.

For reference, for example, an intra prediction mode may include twonon-directional (or non-angular) intra prediction modes and 65directional (or angular) intra prediction modes. The non-directionalintra prediction mode may include a No. 0 (planar) intra prediction modeand a No. 1 DC intra prediction mode. The directional intra predictionmodes may include No. 2 to No. 66 sixty-five intra prediction modes.However, they are examples, and the present disclosure may be applied toa case where the number of intra prediction modes is different.Meanwhile, in some cases, No. 67 intra prediction mode may be furtherused. The No. 67 intra prediction mode may represent a linear model (LM)mode.

FIG. 6 illustrates 65 intra direction modes of a prediction direction.

Referring to FIG. 6, modes may be divided into intra prediction modeshaving horizontal directionality and intra prediction modes havingvertical directionality based on a No. 34 intra prediction mode having aleft-upward diagonal prediction direction. In FIG. 6, H and V mean thehorizontal directionality and the vertical directionality, respectively,and numbers −32˜32 indicate the displacements of a 1/32 unit on a samplegrid position. No. 2 to No. 33 intra prediction modes have horizontaldirectionality, and No. 34 to No. 66 intra prediction modes havevertical directionality. The No. 18 intra prediction mode and the No. 50intra prediction mode indicate a horizontal intra prediction mode and avertical intra prediction mode, respectively. The No. 2 intra predictionmode may be called a left-downward diagonal intra prediction mode, theNo. 34 intra prediction mode may be called a left-upward diagonal intraprediction mode, and the No. 66 intra prediction mode may be called aright-upward diagonal intra prediction mode.

The transformer may perform a secondary transform based on the (primary)transform coefficients and derive (secondary) transform coefficients(step, S520). Whereas the primary transform is a transform from aspatial domain to a frequency domain, the secondary transform may beregarded as a transform from a frequency domain to a frequency domain.The secondary transform may include a non-separable transform. In thiscase, the secondary transform may be called a non-separable secondarytransform (NSST) or mode-dependent non-separable secondary transform(MDNSST). The non-separable secondary transform may represent atransform for generating transform coefficients (or secondary transformcoefficients) for a residual signal by secondary-transform the (primary)transform coefficients derived through the primary transform based on anon-separable transform matrix. Here, the vertical transform and thehorizontal transform are not separately (or independently) applied forthe (primary) transform coefficients based on the non-separabletransform matrix, but the vertical transform and the horizontaltransform may be applied once. In other words, the non-separablesecondary transform may represent a transform method for generatingtransform coefficients (or secondary transform coefficients) bytransforming a vertical component and a horizontal component togetherbased on the non-separable transform matrix, not transforming a verticalcomponent and a horizontal component separately. The non-separablesecondary transform may be applied to a top-left area of a blockincluding (primary) transform coefficients (hereinafter, this may becalled a transform coefficient block or a target block). For example, inthe case that both of a width (W) and a height (H) of the transformcoefficient block are 8 or more, 8×8 non-separable secondary transformmay be applied to top-left 8×8 area of the transform coefficient block(hereinafter, top-left target area). In addition, in the case thateither one of a width (W) and a height (H) of the transform coefficientblock is 8 or smaller while both of a width (W) and a height (H) of thetransform coefficient block are 4 or more and, 4×4 non-separablesecondary transform may be applied to top-left min(8,W)×min(8,H) area ofthe transform coefficient block.

Particularly, for example, in the case that 4×4 input block is used, thenon-separable secondary transform may be performed as below.

The 4×4 input block X may be represented as below.

$\begin{matrix}{X = \begin{bmatrix}X_{00} & X_{01} & X_{02} & X_{03} \\X_{10} & X_{11} & X_{12} & X_{13} \\X_{20} & X_{21} & X_{22} & X_{23} \\X_{30} & X_{31} & X_{32} & X_{33}\end{bmatrix}} & \left\lbrack {{Equation}1} \right\rbrack\end{matrix}$

When the X is represented as a vector form, vector

may be represented as below.

=[X ₀₀ X ₀₁ X ₀₂ X ₀₃ X ₁₀ X ₁₁ X ₁₂ X ₁₃ X ₂₀ X ₂₁ X ₂₂ X ₂₃ X ₃₀ X ₃₁X ₃₂ X ₃₃]^(T)  [Equation 2]

In this case, the non-separable secondary transform may be calculated asbelow.

=T·

  [Equation 3]

Herein,

represents a transform coefficient vector, and T represents 16×16 (non-

separable) transform matrix.

Through Equation 3 above, 16×1 transform coefficient vector

may be derived, and the

may be re-organized with a 4×4 block through a scan order (horizontal,vertical, diagonal,

etc.). However, the calculation is an example, and Hypercube-GivensTransform (HyGT) and the like may be used for calculating thenon-separable secondary transform in order to reduce a calculationcomplexity of the non-separable secondary transform.

Meanwhile, in the non-separable secondary transform, a transform kernel(or transform core, transform type) may be selected in mode dependentmanner. Here, the mode may include an intra prediction mode and/or aninter prediction mode.

As described above, the non-separable secondary transform may beperformed based on 8×8 transform or 4×4 transform determined based on awidth (W) and a height (H) of the transform coefficient block. That is,the non-separable secondary transform may be performed based on 8×8subblock size or 4×4 subblock size. For example, for selecting the modedependent transform kernel, three 35-set of non-separable secondarytransform kernels may be configured for the non-separable secondarytransform for both of 8×8 subblock size and 4×4 subblock size. That is,35 transform sets are configured for 8×8 subblock size and 35 transformsets are configured for 4×4 subblock size. In this case, three 8×8transform kernels may be included in each of 35 transform sets for 8×8subblock size. In this case, three 4×4 transform kernels may be includedin each of 35 transform sets for 4×4 subblock size. However, thetransform subblock size, the number of sets and the number of thetransform kernels in a set are examples, but a size except 8×8 or 4×4may be used, or n sets may be configured, and k transform kernels may beincluded in each set.

The transform set may be called NSST set, and the transform kernel inthe NSST set may be called NSST kernel. A selection of a specific setamong the transform sets may be performed based on an intra predictionmode of a target block (CU or subblock), for example.

In this case, mapping between the 35 transform sets and the intraprediction modes may be represented as the following Table, for example.For reference, in the case that the LM mode is applied to a targetblock, the secondary transform may not be applied to the target block.

TABLE 4 intra mode 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 set 0 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16 intra mode 17 18 19 20 21 22 23 24 2526 27 28 29 30 31 32 33 set 17 18 19 20 21 22 23 24 25 26 27 28 29 30 3132 33 intra mode 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 set34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 intra mode 51 52 5354 55 56 57 58 59 60 61 62 63 64 65 66 67 (LM) set 17 16 15 14 13 12 1110 9 8 7 6 5 4 3 2 NULL

Meanwhile, when it is determined that a specific set is used, one of ktransform kernels in the specific set may be selected through anon-separable secondary transform index. The encoding apparatus mayderive a non-separable secondary transform index indicating a specifictransform kernel based on rate-distortion (RD) check and signal thenon-separable secondary transform index to the decoding apparatus. Thedecoding apparatus may select one of k transform kernels in the specificset based on the non-separable secondary transform index. For example,NSST index value 0 may indicate a first non-separable secondarytransform kernel, NSST index value 1 may indicate a second non-separablesecondary transform kernel, and NSST index value 2 may indicate a thirdnon-separable secondary transform kernel. Alternatively, NSST indexvalue 0 may indicate that a first non-separable secondary transform isnot applied to a target block, and NSST index values 1 to 3 may indicatethe three transform kernels.

Referring to FIG. 5 again, the transformer may perform the non-separablesecondary transform based on the selected transform kernels and obtain(secondary) transform coefficients. As described above, the transformcoefficients may be derived as quantized transform coefficients throughthe quantizer and encoded and signaled to the decoding apparatus andforwarded to the dequantizer/inverse transformer in the encodingapparatus.

Meanwhile, in the case that the secondary transform is omitted, the(primary) transform coefficients, which are an output of the primary(separable) transform, may be derived as quantized transformcoefficients through the quantizer as described above and encoded andsignaled to the decoding apparatus and forwarded to thedequantizer/inverse transformer in the encoding apparatus.

The inverse transformer may perform a series of procedures in a reverseorder of the procedures performed in the transformer. The inversetransformer may receive (dequantized) transform coefficients and derive(primary) transform coefficients by performing secondary (inverse)transform (step, S550) and obtain a residual block (residual samples) byperforming primary (inverse) transform for the (primary) transformcoefficients. Here, the primary transform coefficients may be calledmodified transform coefficients in an aspect of the inverse transformer.The encoding apparatus and the decoding apparatus may generate areconstructed block based the residual block and the predicted block,and based on it, generate a reconstructed picture, as described above.

Meanwhile, as described above, in the case that the secondary transformis omitted, (dequantized) transform coefficients are received, theprimary (separable) transform is performed, and a residual block(residual samples) may be obtained. The encoding apparatus and thedecoding apparatus may generate a reconstructed block based the residualblock and the predicted block, and based on it, generate a reconstructedpicture, as described above.

Meanwhile, the non-separable secondary transform may not be applied to ablock coded with a transform skip mode. For example, in the case that anNSST index for a target CU is signaled and a value of the NSST index isnot 0, the non-separable secondary transform may not be applied to ablock coded with a transform skip mode in the target CU. In addition, inthe case that the target CU including blocks of all components (lumacomponent, chroma component, etc.) is coded with the transform skip modeor in the case that the number of transform coefficients which isnon-zero among the transform coefficients for the target CU is smallerthan 2, the NSST index may not be signaled. A detailed coding process ofa transform coefficient is as below.

FIG. 7a and FIG. 7b are flowcharts illustrating a coding process of atransform coefficient according to an embodiment.

Each step shown in FIG. 7a and FIG. 7b may be performed by the encodingapparatus 100 or the decoding apparatus 300 shown in FIG. 1 and FIG. 3,respectively, and more particularly, performed by the entropy encoder130 shown in FIG. 1 and the entropy decoder 310 shown in FIG. 3,respectively. Accordingly, the detailed description overlapped with thecontents described in FIG. 1 or FIG. 3 is omitted or briefly described.

In this disclosure, a term or a sentence is used for defining specificinformation or concept. For example, in this disclosure, “a flagindicating whether at least one non-zero transform coefficient ispresent among transform coefficients for a target block” indicates acbf. However, since “cbf” may be replaced by other various terms such ascoded_block_flag and the like, when interpreting a term or a sentenceused for defining specific information or concept throughout thedisclosure, the term or the sentence should not be interpreted in alimited manner to the title, but it is required to interpret the term orthe sentence by concentrating on various operations, functions andeffects according to the meaning of the term.

FIG. 7a illustrates an encoding process of a transform coefficient.

The encoding apparatus 100 according to an embodiment may determinewhether a flag indicating whether at least one non-zero transformcoefficient is present among transform coefficients for a target blockindicates 1 (step, S700). In the case that the flag indicating whetherat least one non-zero transform coefficient is present among transformcoefficients for a target block indicates 1, at least one non-zerotransform coefficient may be present among transform coefficients for atarget block. On the other hand, in the case that the flag indicatingwhether at least one non-zero transform coefficient is present amongtransform coefficients for a target block indicates 0, all the transformcoefficients for a target block may indicate 0.

The flag indicating whether at least one non-zero transform coefficientis present among transform coefficients for a target block may berepresented by a cbf flag. The cbf flag may includecbf_luma[x0][y0][trafoDepth] flag for a luma block andcbf_cb[x0][y0][trafoDepth] and cbf_cr[x0][y0][trafoDepth] flags for achroma block. Here, the array indices x0 and y0 may mean positions oftop-left luma/chroma samples of a target block for top-left luma/chromasamples of a current picture, and the array index trafoDepth may mean alevel in which a coding block is partitioned for the purpose oftransform coding. The blocks of which trafoDepth indicates 0 maycorrespond to a coding block, and in the case that a coding block and atransform block are defined identically, trafoDepth may be regarded as0.

In the case that the flag indicating whether at least one non-zerotransform coefficient is present among transform coefficients for atarget block indicates 1 in step S700, the encoding apparatus 100according to an embodiment may encode information for transformcoefficients for a target block (step, S710).

The information for transform coefficients for a target block mayinclude at least one of information for a position of the last transformcoefficient, which is non-zero, group flag information indicatingwhether a non-zero transform coefficient is included in a sub-group of atarget block and information for a reduced coefficient, for example.Detailed description for the information will be described below.

The encoding apparatus 100 according to an embodiment may determinewhether it corresponds to a condition for perform NSST (step, S720).More particularly, the encoding apparatus 100 may determine whether itcorresponds to a condition for encoding an NSST index. In this case, theNSST index may be called a transform index, for example.

In the case that the encoding apparatus 100 according to an embodimentdetermines that it corresponds to a condition for perform NSST in stepS720, the encoding apparatus 100 may encode the NSST index (step, S730).More particularly, in the case that the encoding apparatus 100 accordingto an embodiment determines that it corresponds to a condition forencoding the NSST index, the encoding apparatus 100 may encode the NSSTindex.

In the case that the flag indicating whether at least one non-zerotransform coefficient is present among transform coefficients for atarget block indicates 0 in step S700, the encoding apparatus 100according to an embodiment may omit the operations according to stepsS710, S720 and S730.

In addition, in the case that the encoding apparatus 100 according to anembodiment may determine whether it does not correspond to a conditionfor perform NSST in step S720, the encoding apparatus 100 may omit theoperation according to step S730.

FIG. 7b illustrates a decoding process of a transform coefficient.

The decoding apparatus 300 according to an embodiment may determinewhether a flag indicating whether at least one non-zero transformcoefficient is present among transform coefficients for a target blockindicates 1 (step, S740). In the case that the flag indicating whetherat least one non-zero transform coefficient is present among transformcoefficients for a target block indicates 1, at least one non-zerotransform coefficient may be present among transform coefficients for atarget block. On the other hand, in the case that the flag indicatingwhether at least one non-zero transform coefficient is present amongtransform coefficients for a target block indicates 0, all the transformcoefficients for a target block may indicate 0.

In the case that the flag indicating whether at least one non-zerotransform coefficient is present among transform coefficients for atarget block indicates 1 in step S740, the decoding apparatus 300according to an embodiment may encode information for transformcoefficients for a target block (step, S750).

The decoding apparatus 300 according to an embodiment may determinewhether it corresponds to a condition for perform NSST (step, S760).More particularly, the decoding apparatus 300 may determine whether itcorresponds to a condition for decoding an NSST index from a bitstream.

In the case that the decoding apparatus 300 according to an embodimentdetermines that it corresponds to a condition for perform NSST in stepS760, the decoding apparatus 300 may decode the NSST index (step, S770).

In the case that the flag indicating whether at least one non-zerotransform coefficient is present among transform coefficients for atarget block indicates 0 in step S740, the decoding apparatus 300according to an embodiment may omit the operations according to stepsS750, S760 and S770.

In addition, in the case that the decoding apparatus 300 according to anembodiment may determine whether it does not correspond to a conditionfor perform NSST in step S760, the decoding apparatus 300 may omit theoperation according to step S770.

As described above, it may degrade a coding efficiency to signal theNSST index when NSST is not performed as described above. In addition,depending on a specific condition, a method for differentiating a codingmethod of the NSST index may improve overall coding efficiency of animage. Accordingly, the present disclosure proposes various NSST indexcoding methods.

For example, based on a specific condition, a range of the NSST indexmay be determined. In other words, based on a specific condition, arange of a value of the NSST index may be determined. Particularly,based on a specific condition, a maximum value of the NSST index may bedetermined.

For example, based on a block size, a range of a value of the NSST indexmay be determined. Here, the block size may be defined as a minimum (W,H). The W may represent a width, and H may represent a height. In thiscase, through a comparison between a width of a target block and the Wand a comparison between a height of the target block and the minimum H,a range of a value of the NSST index may be determined.

Alternatively, the block size may be defined as the number of samples ina block (W*H). In this case, through a comparison between the number ofsamples of a target block W*H and a specific value, a range of a valueof the NSST index may be determined.

Alternatively, for example, based on a shape of a block, that is, ablock type, a range of a value of the NSST index may be determined.Here, the block type may be defined as a square block or a non-squareblock. In this case, a range of a value of the NSST index may bedetermined based on whether a target block is a square block or anon-square block.

Alternatively, the block type may be defined as a ratio of a long side(long side between a width and a height) and a short side of a block. Inthis case, through a comparison between a ratio of a long side and ashort side of a block and a preconfigured threshold value (e.g., 2 or3), a range of a value of the NSST index may be determined. Here, theratio may represent a value of the short side divided by the long side.For example, in the case that a width of the target block is longer thana height, through a comparison between the value of the width divided bythe height and the preconfigured threshold value, a range of a value ofthe NSST index may be determined. In addition, in the case that a heightof the target block is longer than a width, through a comparison betweenthe value of the height divided by the width and the preconfiguredthreshold value, a range of a value of the NSST index may be determined.

Alternatively, as an example, based on an intra prediction mode appliedto a block, a range of a value of the NSST index may be determined. Forexample, based on whether an intra prediction mode applied to the targetblock is a non-directional intra prediction mode or a directional intraprediction mode, a range of a value of the NSST index may be determined.

Alternatively, as another example, based on whether an intra predictionmode applied to the target block is an intra prediction mode included incategory A or category B, a range of a value of the NSST index may bedetermined. Here, as an example, the category A may include No. 2 intraprediction mode, No. 10 intra prediction mode, No. 18 intra predictionmode, No. 26 intra prediction mode, No. 34 intra prediction mode, No. 42intra prediction mode, No. 50 intra prediction mode, No. 58 intraprediction mode and No. 66 intra prediction mode, and the category B mayinclude intra prediction modes except the intra prediction modesincluded in the category A. The intra prediction modes included in thecategory A may be preconfigured, and it may be preconfigured that thecategory A and the category B include intra prediction modes which aredifferent from the example described above.

Alternatively, as another example, based on AMT factor of a block, arange of a value of the NSST index may be determined. The AMT factor mayalso be represented as MTS factor.

For example, the AMT factor may be defined as the AMT flag describedabove. In this case, based on a value of the AMT flag of the targetblock, a range of a value of the NSST index may be determined.

Alternatively, the AMT factor may be defined as the AMT vertical flagand/or the AMT horizontal flag described above. In this case, based on avalue of the AMT vertical flag and/or the AMT horizontal flag of thetarget block, a range of a value of the NSST index may be determined.

Alternatively, the AMT factor may be defined as a transform kernel whichis applied in the multi-core transform. In this case, based on thetransform kernel applied in the multi-core transform of the targetblock, a range of a value of the NSST index may be determined.

Alternatively, as another example, based on a component of a block, arange of a value of the NSST index may be determined. For example, arange of a value of the NSST index for a luma block of the target blockand a range of a value of the NSST index for a chroma block of thetarget block may be differently applied.

Meanwhile, through a combination of the specific conditions describedabove, a range of a value of the NSST index may be determined.

A range of a value of the NSST index determined based on the specificcondition, that is, a maximum value of the NSST index may be configuredin various manners.

For example, based on the specific condition, the maximum value of theNSST index may be determined to be R1, R2 or R3. Particularly, in thecase that the specific condition corresponds to category A, the maximumvalue of the NSST index may be derived as R1. In the case that thespecific condition corresponds to category B, the maximum value of theNSST index may be derived as R2. In the case that the specific conditioncorresponds to category C, the maximum value of the NSST index may bederived as R3.

R1 for the category A, R2 for the category B and R3 for the category Cmay be derived as represented in the following Table.

TABLE 5 Maximum NSST Idx Category A R1 Category B R2 Category C R3 . . .. . .

R1, R2 and R3 may be preconfigured. For example, a relationship amongR1, R2 and R3 may be derived as represented in the following Equation.

0<=R1<R2<R3< . . .   [Equation 4]

Referring to Equation 4, R1 may be equal to or greater than 0, R2 may begreater than R1, and R3 may be greater than R2. Meanwhile, in the casethat R2 is 0, and in the case that the maximum value of the NSST indexis determined to be R1, the NSST index may not be signaled, and a valueof the NSST index may be inferred as 0.

In addition, in the present disclosure, an implicit NSST index codingmethod is proposed.

Generally, in the case that NSST is applied, a distribution of non-zerotransform coefficients among transform coefficients may be changed.Particularly, in the case that reduced secondary transform (RST) to asecondary transform is used in a specific condition, an NSST index maynot be coded.

Here, the RST may represent a secondary transform that uses a reducedtransform matrix as a non-separable transform matrix, and the reducedtransform matrix may be determined such that an N dimensional vector ismapped to a R dimensional vector located in different space. Here, R issmaller than N. The N may mean a square of a length of a side of a blockto which transform is applied or total number of transform coefficientscorresponding to a block to which transform is applied. A reduced factormay mean R/N value. The reduced factor may be called by various termssuch as a simplification factor, a reduction factor, simplified factor,simple factor, and the like. Meanwhile, the R may be called a reducedcoefficient, but in some cases, the reduced factor may mean the R. Inaddition, in some cases, the reduced factor may mean N/R value.

A size of the reduced transform matrix according to an embodiment may beR×N which is smaller than a size N×N of a common transform matrix andmay be defined as represented in Equation 5 below.

$\begin{matrix}{T_{RxN} = \begin{bmatrix}t_{11} & t_{12} & t_{13} & \ldots & t_{1N} \\t_{21} & t_{22} & t_{23} & & t_{2N} \\ & \vdots & & \ddots & \vdots \\t_{R1} & t_{R2} & t_{R3} & \ldots & t_{RN}\end{bmatrix}} & \left\lbrack {{Equation}5} \right\rbrack\end{matrix}$

When a reduced transform matrix T_(R×N) is multiplied to the transformcoefficients to which a primary transform of a target block is applied,(secondary) transform coefficients for the target block may be derived.

In the case that the RST is applied, since the reduced transform matrixof R×N size is applied to the secondary transform, transformcoefficients from R+1 to N may be 0 implicitly. In other words, in thecase that the RST is applied, and the transform coefficient of thetarget block is derived, transform coefficients from R+1 to N may be 0.Here, the transform coefficients from R+1 to N may represent transformcoefficients from R+1^(th) transform coefficient to N^(th) transformcoefficient. Particularly, an array of transform coefficients of thetarget block may be described as below.

FIG. 8 is a diagram for describing an array of transform coefficientsbased on a target block according to an embodiment of the presentdisclosure. Hereinafter, the description for the transform in FIG. 8described below may also be applied to an inverse transform in the samemanner. For a target block (or residual block, 800), NSST (an example ofa secondary transform) based on a primary transform and a reducedtransform may be performed. In an example, the 16×16 block shown in FIG.8 may represent the target block 800, and 4×4 blocks denoted by A to Pmay represent sub-groups of the target block 800. The primary transformmay be performed in the whole range of the target block 800. After theprimary transform is performed, NSST may be applied to 8×8 block(hereinafter, top-left target area) constructed by sub-groups A, B, Eand F. In this case, when NSST based on the reduced transform isperformed, since only R NSST transform coefficients (herein, R means areduced coefficient, and R is smaller than N) are derived, each of theNSST transform coefficients in the R+1^(th) to N^(th) range may bedetermined to be 0. In the case that R is 16, for example, 16 transformcoefficients to which NSST based on the reduced transform is performed,which is derived, may be allocated to each of the blocks included insub-group A, which is a top-left 4×4 block included in the top-lefttarget area of the target block 800, and transform coefficient 0 may beallocated to each of N−R blocks, that is, 64−16=48 groups, included insub-group B, E and F. The primary transform coefficients to which NSSTbased on the reduced transform is not performed may be allocated to eachof the blocks included in sub-groups C, D, G, H, I, J, K, L, M, N, O andP.

Accordingly, in the case that transform coefficients from R+1 to N arescanned and any one non-zero transform coefficient is derived, it may bedetermined that the RST is not applied, and a value of the NSST indexmay be 0 implicitly without any separate signaling. That is, in the casethat transform coefficients from R+1 to N are scanned and any onenon-zero transform coefficient is derived, the RST may not be applied,and a value of the NSST index may be derived as 0 without any separatesignaling.

FIG. 9 illustrates an example of scanning transform coefficients fromR+1 to N.

Referring to FIG. 9, a size of a target block to which transform isapplied may be 64×64, and R=16 (i.e., R/N=16/64=1/4). That is, FIG. 9shows a top-left target area of the target block. A reduced transformmatrix of 16×64 size may be applied to a secondary transform for 64samples of the top-left target area of the target block. In this case,when the RST is applied to the top-left target area, a value oftransform coefficients from 17 to 64 (N) needs to be 0. In other words,in the case that any one non-zero transform coefficient is derived from17 to 64 transform coefficients of the target block, the RST may not beapplied, and a value of the NSST index may be derived as 0 without anyseparate signaling. Accordingly, the decoding apparatus may decode thetransform coefficient of the target block and scan the transformcoefficients from 17 to 64 among the decoded transform coefficients. Inthe case that a non-zero transform coefficient is derived, the decodingapparatus may derive a value of the NSST index as 0 without any separatesignaling. Meanwhile, in the case that a non-zero transform coefficientis not existed among 17 to 64 transform coefficients, the decodingapparatus may receive and decode the NSST index.

FIG. 10a and FIG. 10b are flowcharts illustrating a coding process of anNSST index according to an embodiment.

FIG. 10a shows an encoding process of an NSST index.

The encoding apparatus may encode a transform coefficient for a targetblock (step, S1000). The encoding apparatus may perform entropy encodingfor quantized transform coefficients. The entropy encoding may includean encoding method such as exponential Golomb, context-adaptive variablelength coding (CAVLC), context-adaptive binary arithmetic coding(CABAC), and the like.

The encoding apparatus may determine whether an (explicit) NSST indexfor the target block is coded (step, S1010). Here, the (explicit) NSSTindex may represent the NSST index transferred to the decodingapparatus. That is, the encoding apparatus may determine whether togenerate the NSST index which is signaled. In other words, the encodingapparatus may determine whether to allocate a bit for a syntax elementfor the NSST index. As described in the embodiment, in the case that thedecoding apparatus may derived a value of the NSST index without theNSST index being signaled, the encoding apparatus may not code the NSSTindex. The detailed process of determining whether the NSST index iscoded is as described below.

In the case that it is determined that the (explicit) NSST index iscoded, the encoding apparatus may encode the NSST index (step, S1020).

FIG. 10b shows a decoding process of an NSST index.

The decoding apparatus may decode a transform coefficient for a targetblock (step, S1030).

The decoding apparatus may determine whether an (explicit) NSST indexfor the target block is coded (step, S1040). Here, the (explicit) NSSTindex may represent the NSST index signaled from the encoding apparatus.As described in the embodiment, in the case that the decoding apparatusmay derived a value of the NSST index without the NSST index beingsignaled, the NSST index may not be signaled from the encodingapparatus. The detailed process of determining whether the NSST index iscoded is as described below.

In the case that it is determined that the (explicit) NSST index iscoded, the decoding apparatus may encode the NSST index (step, S1040).

FIG. 11 illustrates an example of determining whether an NSST index iscoded.

The encoding apparatus/decoding apparatus may determine whether itcorresponds to a condition of coding an NSST index for a target block(step, S1100). For example, in the case that cbf flag for the targetblock indicates 0, the encoding apparatus/decoding apparatus maydetermine not to code the NSST index for the target block.Alternatively, in the case that the target block is coded with atransform skip mode or the number of non-zero transform coefficientsamong the transform coefficients for the target block is smaller than apreconfigured threshold value, the encoding apparatus/decoding apparatusmay determine not to code the NSST index for the target block. Forexample, the preconfigured threshold value may be 2.

In the case that it corresponds to a condition of coding the NSST indexfor the target block, the encoding apparatus/decoding apparatus may scantransform coefficients from R+1 to N (step, S1110). The transformcoefficients from R+1 to N may represent transform coefficients fromR+1^(th) to N^(th) on a scan order among the transform coefficients.

The encoding apparatus/decoding apparatus may determine whether anon-zero transform coefficient is derived among the transformcoefficients from R+1 to N (step, S1120). In the case that a non-zerotransform coefficient is derived among the transform coefficients fromR+1 to N, the encoding apparatus/decoding apparatus may determine not tocode the NSST index for the target block. In this case, encodingapparatus/decoding apparatus may derive the NSST index for the targetblock as 0. In other words, for example, in the case that the NSST indexof which value is 0 indicates that NSST is not applied, the encodingapparatus/decoding apparatus may not perform NSST for the top-lefttarget area of the target block.

Meanwhile, in the case that a non-zero transform coefficient is notderived among the transform coefficients from R+1 to N, the encodingapparatus may encode the NSST index for the target block, and thedecoding apparatus may decode the NSST index for the target block.

Meanwhile, it may be proposed a method for components (a luma component,a chroma Cb component and a chroma Cr component) to use a common NSSTindex.

For example, the same NSST index may be used for a chroma Cb componentof the target block and a chroma Cr component of the target block. Inaddition, as another example, the same NSST index may be used for a lumacomponent of the target block, a chroma Cb component of the target blockand a chroma Cr component of the target block.

In the case that two or three components of the target block use thesame NSST index, the encoding apparatus may scan transform coefficientsfrom R+1 to N of all components (a luma block, a chroma Cb block and achroma Cr block of the target block), and in the case that at least onenon-zero coefficient is derived, the encoding apparatus may not encodethe NSST index but derive a value of the NSST index as 0. In addition,the decoding apparatus may scan transform coefficients from R+1 to N ofall components (a luma block, a chroma Cb block and a chroma Cr block ofthe target block), and in the case that at least one non-zerocoefficient is derived, the decoding apparatus may not decode the NSSTindex but derive a value of the NSST index as 0.

FIG. 12 illustrates an example of scanning transform coefficients fromR+1 to N for all components of a target block.

Referring to FIG. 12, a size of a luma block, a chroma Cb block and achroma Cr block of a target block to which transform is applied may be64×64, and R=16 (i.e., R/N=16/64=1/4). That is, FIG. 12 shows a top-lefttarget area of the luma block, a top-left target area of the chroma Cbblock and a top-left target area of the chroma Cr block. Accordingly, areduced transform matrix of 16×64 size may be applied to a secondarytransform for each of 64 samples of a top-left target area of the lumablock, a top-left target area of the chroma Cb block and a top-lefttarget area of the chroma Cr block. In this case, when the RST isapplied to a top-left target area of the luma block, a top-left targetarea of the chroma Cb block and a top-left target area of the chroma Crblock, a value of transform coefficients from 17 to 64 (N) needs to be0. In other words, in the case that any one non-zero transformcoefficient is derived from 17 to 64 transform coefficients of thetarget block, the RST may not be applied, and a value of the NSST indexmay be derived as 0 without any separate signaling. Accordingly, thedecoding apparatus may decode the transform coefficient of allcomponents of the target block and scan the transform coefficients from17 to 64 of the luma block, the chroma Cb block and the chroma Cr blockamong the decoded transform coefficients. In the case that a non-zerotransform coefficient is derived, the decoding apparatus may derive avalue of the NSST index as 0 without any separate signaling. Meanwhile,in the case that a non-zero transform coefficient is not existed among17 to 64 transform coefficients, the decoding apparatus may receive anddecode the NSST index. The NSST index may be used for an index for theluma block, the chroma Cb block and the chroma Cr block.

In addition, in the present disclosure, it may be proposed a method forsignaling an NSST index indicator in a higher level. NSST_Idx_indicatormay represent a syntax element for the NSST index indicator. Forexample, the NSST index indicator may be coded with Coding Tree Unit(CTU) level, and the NSST index indicator may represent whether NSST isapplied to a target CTU. That is, the NSST index indicator may representwhether NSST is available for a target CTU. Particularly, in the casethat the NSST index indicator is enabled for the target CTU (in the casethat NSST is available for the target CTU), that is, in the case that avalue of the NSST index indicator is 1, an NSST index for a CU or a TUincluded in the target CTU may be coded. In the case that the NSST indexindicator is disabled for the target CTU (in the case that NSST isunavailable for the target CTU), that is, in the case that a value ofthe NSST index indicator is 0, an NSST index for a CU or a TU includedin the target CTU may not be coded. Meanwhile, the NSST index indicatormay be coded with a CTU level as described above or coded with a samplegroup level of different arbitrary size. For example, the NSST indexindicator may be coded with a Coding Unit (CU) level.

FIG. 13 schematically illustrates a video encoding method by an encodingapparatus according to the present disclosure. The method shown in FIG.13 may be performed by the encoding apparatus shown in FIG. 1.Particularly, for example, step S1300 of FIG. 13 may be performed by thesubtractor the encoding apparatus, step S1310 may be performed by thetransformer of the encoding apparatus, and steps S1320 to S1330 may beperformed by the entropy encoder of the encoding apparatus. In addition,although it is not shown, the process of deriving a prediction samplemay be performed by the predictor of the encoding apparatus.

The encoding apparatus derives residual samples of a target block (step,S1300). For example, the encoding apparatus may determine whether toperform an inter prediction or an intra prediction for the target blockand determine a specific inter prediction mode or a specific intraprediction mode based on RD cost. According to the encoding apparatusmay derive prediction samples for the target block and derive theresidual samples through a summation of original samples for the targetblock and the prediction samples.

The encoding apparatus derives transform coefficients of the targetblock by performing transform for the residual samples (step, S1310).The encoding apparatus may determine whether to apply NSST for thetarget block.

In the case that the NSST is applied to the target block, the encodingapparatus may derive modified transform coefficients by performing acore transform for the residual samples and derive the transformcoefficients of the target block by performing NSST for the modifiedtransform coefficients located in a top-left target area of the targetblock based on a reduced transform matrix. The modified transformcoefficients except the modified transform coefficients located in thetop-left target area of the target block may be derived as the transformcoefficients of the target block without any change. A size of thereduced transform matrix may be R×N. Herein, the N may be the number ofsamples in the top-left target area, and the R may be a reducedcoefficient. The R may be smaller than N.

Particularly, the core transform for the residual samples may beperformed as below. The encoding apparatus may determine whether toapply Adaptive Multiple core Transform (AMT) for the target block. Inthis case, an AMT flag may be generated, which represents whether theAdaptive Multiple core Transform is applied to the target block. In thecase that the AMT is not applied to the target block, the encodingapparatus may derive DCT type 2 as a transform kernel for the targetblock and derive the modified transform coefficients by performingtransform for the residual samples based on DCT type 2.

In the case that the AMT is applied to the target block, the encodingapparatus may configure a transform subset for a horizontal transformkernel and a transform subset for a vertical transform kernel, derive ahorizontal transform kernel and a vertical transform kernel based on thetransform subsets, and derive modified transform coefficients byperforming transform for the residual samples based on the horizontaltransform kernel and the vertical transform kernel. Here, the transformsubset for the horizontal transform kernel and the transform subset forthe vertical transform kernel may include DCT type 2, DCT type 7, DCTtype 8 and/or DCT type 1 as a candidate. In addition, transform indexinformation may be generated, and the transform index information mayinclude an AMT horizontal flag indicating the horizontal transformkernel and an AMT vertical flag indicating the vertical transformkernel. Meanwhile, the transform kernel may be called a transform typeor a transform core.

Meanwhile, in the case that the NSST is not applied to the target block,the encoding apparatus may derive the transform coefficients of thetarget block by performing a core transform for the residual samples.

Particularly, the core transform for the residual samples may beperformed as below. The encoding apparatus may determine whether toapply Adaptive Multiple core Transform (AMT) for the target block. Inthis case, an AMT flag may be generated, which represents whether theAdaptive Multiple core Transform is applied to the target block. In thecase that the AMT is not applied to the target block, the encodingapparatus may derive DCT type 2 as a transform kernel for the targetblock and derive the modified transform coefficients by performingtransform for the residual samples based on DCT type 2.

In the case that the AMT is applied to the target block, the encodingapparatus may configure a transform subset for a horizontal transformkernel and a transform subset for a vertical transform kernel, derive ahorizontal transform kernel and a vertical transform kernel based on thetransform subsets, and derive modified transform coefficients byperforming transform for the residual samples based on the horizontaltransform kernel and the vertical transform kernel. Here, the transformsubset for the horizontal transform kernel and the transform subset forthe vertical transform kernel may include DCT type 2, DCT type 7, DCTtype 8 and/or DCT type 1 as a candidate. In addition, transform indexinformation may be generated, and the transform index information mayinclude an AMT horizontal flag indicating the horizontal transformkernel and an AMT vertical flag indicating the vertical transformkernel. Meanwhile, the transform kernel may be called a transform typeor a transform core.

The encoding apparatus determines whether to encode the NSST index(step, S1320).

As an example, the encoding apparatus may scan from R+1^(th) to N^(th)transform coefficients among transform coefficients of the target block.In the case that a non-zero transform coefficient is included in theR+1^(th) to N^(th) transform coefficients, the encoding apparatus maydetermine not to encode the NSST index. Herein, the N may be the numberof samples in the top-left target area, and the R may be a reducedcoefficient. The R may be smaller than N. The N may be derived as amultiplication of a width and a height of the top-left target area.

In addition, in the case that a non-zero transform coefficient is notincluded in the R+1^(th) to N^(th) transform coefficients, the encodingapparatus may determine to encode the NSST index. In this case,information for the transform coefficients may include a syntax elementfor the NSST index. That is, the syntax element for the NSST index maybe encoded. In other words, a bit for the syntax element for the NSSTindex may be allocated.

Meanwhile, the encoding apparatus may determine whether it correspondsto a condition that the NSST is available to be performed. In the casethat the NSST is available to be performed, the encoding apparatus maydetermine to encode the NSST index for the target block. For example,from a bitstream, an NSST index indicator for a target CTU including thetarget block may be generated, and the NSST index indicator mayrepresent whether NSST is applied to the target CTU. In the case that avalue of the NSST index indicator is 1, the encoding apparatus maydetermine to encode the NSST index for the target block, and in the casethat a value of the NSST index indicator is 0, the encoding apparatusmay determine not to encode the NSST index for the target block. Asdescribed in the example above, the NSST index indicator may be signaledwith a CTU level, or the NSST index indicator may be signaled with a CUlevel or other higher level.

In addition, the NSST index may be used for a plurality of components ofthe target block.

For example, the NSST index may be used for inverse transform fortransform coefficients of a luma block of the target block, transformcoefficients of a chroma Cb block of the target block and transformcoefficients of a chroma Cr block of the target block. In this case,R+1^(th) to N^(th) transform coefficients of the luma block, R+1^(th) toN^(th) transform coefficients of the chroma Cb block and R+1^(th) toN^(th) transform coefficients of the chroma Cr block may be scanned. Inthe case that a non-zero transform coefficient is included in thescanned transform coefficients, it may be determined that the NSST indexis not encoded. In the case that a non-zero transform coefficient is notincluded in the scanned transform coefficients, it may be determinedthat the NSST index is encoded. In this case, information for thetransform coefficients may include a syntax element for the NSST index.That is, the syntax element for the NSST index may be encoded. In otherwords, a bit for the syntax element for the NSST index may be allocated.

As another example, the NSST index may be used for inverse transform fortransform coefficients of a luma block of the target block and transformcoefficients of a chroma Cb block of the target block. In this case,R+1^(th) to N^(th) transform coefficients of the luma block and R+1^(th)to N^(th) transform coefficients of the chroma Cb block may be scanned.In the case that a non-zero transform coefficient is included in thescanned transform coefficients, it may be determined that the NSST indexis not encoded. In the case that a non-zero transform coefficient is notincluded in the scanned transform coefficients, it may be determinedthat the NSST index is encoded. In this case, information for thetransform coefficients may include a syntax element for the NSST index.That is, the syntax element for the NSST index may be encoded. In otherwords, a bit for the syntax element for the NSST index may be allocated.

As another example, the NSST index may be used for inverse transform fortransform coefficients of a luma block of the target block and transformcoefficients of a chroma Cr block of the target block. In this case,R+1^(th) to N^(th) transform coefficients of the luma block and R+1^(th)to N^(th) transform coefficients of the chroma Cr block may be scanned.In the case that a non-zero transform coefficient is included in thescanned transform coefficients, it may be determined that the NSST indexis not encoded. In the case that a non-zero transform coefficient is notincluded in the scanned transform coefficients, it may be determinedthat the NSST index is encoded. In this case, information for thetransform coefficients may include a syntax element for the NSST index.That is, the syntax element for the NSST index may be encoded. In otherwords, a bit for the syntax element for the NSST index may be allocated.

Meanwhile, based on a specific condition, a range of the NSST index maybe derived. For example, based on the specific condition, a maximumvalue of the NSST index may be derived, and the range may be derivedfrom 0 to the derived maximum value. The value of the derived NSST indexmay be included in the range.

For example, based on a size of the target block, a range of the NSSTindex may be derived. Particularly, a minimum width and a minimum heightmay be preconfigured. Based on the minimum width and a width of thetarget block, the minimum height and a height of the target block, arange of the NSST index may be derived. In addition, based on a specificvalue or a sample number of the target block, a range of the NSST indexmay be derived. The sample number may be a value of multiplying a widthand a height of the target block, and the specific value may bepreconfigured.

In addition, as another example, based on a type of the target block, arange of the NSST index may be derived. Particularly, a range of theNSST index may be derived based on whether the target block is anon-square block. In addition, a range of the NSST index may be derivedbased on a ratio between a width and a height of the target block and aspecific value. The ratio between a width and a height of the targetblock may be a value of a long side between the width and the height ofthe target block divided by a short side, and the specific value may bepreconfigured.

In addition, as another example, based on an intra prediction mode ofthe target block, a range of the NSST index may be derived.Particularly, a range of the NSST index may be derived based on whetherthe intra prediction mode of the target block is a non-directional intraprediction mode or a directional intra prediction mode. In addition,based on whether an intra prediction mode of the target block is anintra prediction mode included in category A or category B, a range of avalue of the NSST index may be determined. Here, the intra predictionmode included in category A and the intra prediction mode included incategory B may be preconfigured. As an example, the category A mayinclude No. 2 intra prediction mode, No. 10 intra prediction mode, No.18 intra prediction mode, No. 26 intra prediction mode, No. 34 intraprediction mode, No. 42 intra prediction mode, No. 50 intra predictionmode, No. 58 intra prediction mode and No. 66 intra prediction mode, andthe category B may include intra prediction modes except the intraprediction modes included in the category A.

In addition, as another example, based on information for a coretransform of the target block, a range of the NSST index may be derived.For example, a range of the NSST index may be derived based on an AMTflag that represents whether Adaptive Multiple core Transform (AMT) isapplied. In addition, a range of the NSST index may be derived based onan AMT horizontal flag indicating a horizontal transform kernel and anAMT vertical flag indicating a vertical transform kernel.

Meanwhile, in the case that a value of the NSST index is 0, the NSSTindex may indicate that NSST is not applied to the target block.

The encoding apparatus encodes information for transform coefficients(step, S1330). The information for transform coefficients may includeinformation for a size, a position, and the like of the transformcoefficients. In addition, as described above, the information fortransform coefficients may further include the NSST index, the transformindex information and/or the AMT flag. Image information including theinformation for transform coefficients may be output in a bitstreamformat. In addition, the image information may further include the NSSTindex indicator and/or prediction information. The predictioninformation are information related to the prediction process and mayinclude prediction mode information and information for motioninformation (e.g., for the case that inter prediction is applied).

The output bitstream may be forwarded to the decoding apparatus througha storage medium or a network.

FIG. 14 schematically illustrates an encoding apparatus performing avideo encoding method according to the present disclosure. The methodshown in FIG. 13 may be performed by the encoding apparatus shown inFIG. 14. Particularly, for example, the adder of the encoding apparatusmay perform step S1300 of FIG. 13, the transformer of the encodingapparatus may perform step S1310, and the entropy encoder of theencoding apparatus may perform steps S1320 to S1330. In addition,although it is not shown, the process of deriving a prediction samplemay be performed by the predictor of the encoding apparatus.

FIG. 15 schematically illustrates a video decoding method by a decodingapparatus according to the present disclosure. The method shown in FIG.15 may be performed by the decoding apparatus shown in FIG. 3.Particularly, for example, steps S1500 to S1510 of FIG. 15 may beperformed by the entropy decoder the decoding apparatus, step S1520 maybe performed by the inverse transformer of the decoding apparatus, andstep S1530 may be performed by the adder of the decoding apparatus. Inaddition, although it is not shown, the process of deriving a predictionsample may be performed by the predictor of the decoding apparatus.

The decoding apparatus derives transform coefficients of the targetblock from the bitstream (step, S1500). The decoding apparatus maydecode the information for the transform coefficients of the targetblock received from the bitstream and derive transform coefficients ofthe target block. The received information for the transformcoefficients of the target block may be represented as residualinformation.

Meanwhile, the transform coefficients of the target block may includetransform coefficients of a luma block of the target block, transformcoefficients of a chroma Cb block of the target block and transformcoefficients of a chroma Cr block of the target block.

The decoding apparatus derives a Non-Separable Secondary Transform(NSST) index for the target block (step, S1510).

As an example, the decoding apparatus may scan R+1^(th) to N^(th)transform coefficients among the transform coefficients of the targetblock. In the case that a non-zero transform coefficient is included inthe R+1^(th) to N^(th) transform coefficients, the decoding apparatusmay derive a value of the NSST index as 0. Here, the N is a samplenumber of a top-left target area of the target block, and the R is areduced coefficient, and the R may be smaller than the N. The N may bederived as a multiplication of a width and a height of the top-lefttarget area.

In addition, in the case that a non-zero transform coefficient is notincluded in the R+1^(th) to N^(th) transform coefficients, the decodingapparatus may parse a syntax element for the NSST index and derive avalue of the NSST index. That is, in the case that a non-zero transformcoefficient is not included in the R+1^(th) to N^(th) transformcoefficients, the bitstream may include the syntax element for the NSSTindex, and the decoding apparatus may parse the syntax element for theNSST index and derive a value of the NSST index.

Meanwhile, the decoding apparatus may determine whether it correspondsto a condition that the NSST is available to be performed. In the casethat the NSST is available to be performed, the decoding apparatus mayderive the NSST index for the target block. For example, from abitstream, an NSST index indicator for a target CTU including the targetblock may be signaled, and the NSST index indicator may representwhether NSST is enabled in the target CTU. In the case that a value ofthe NSST index indicator is 1, the decoding apparatus may derive theNSST index for the target block, and in the case that a value of theNSST index indicator is 0, the decoding apparatus may not derive theNSST index for the target block. As described in the example above, theNSST index indicator may be signaled with a CTU level, or the NSST indexindicator may be signaled with a CU level or other higher level.

In addition, the NSST index may be used for a plurality of components ofthe target block.

For example, the NSST index may be used for inverse transform fortransform coefficients of a luma block of the target block, transformcoefficients of a chroma Cb block of the target block and transformcoefficients of a chroma Cr block of the target block. In this case,R+1^(th) to N^(th) transform coefficients of the luma block, R+1^(th) toN^(th) transform coefficients of the chroma Cb block and R+1^(th) toN^(th) transform coefficients of the chroma Cr block may be scanned. Inthe case that a non-zero transform coefficient is included in thescanned transform coefficients, a value of the NSST index may be derivedas 0. In the case that a non-zero transform coefficient is not includedin the scanned transform coefficients, the bitstream may include asyntax element for the NSST index, and a value of the NSST index may bederived by parsing the syntax element for the NSST index receivedthrough a bitstream.

As another example, the NSST index may be used for inverse transform fortransform coefficients of a luma block of the target block and transformcoefficients of a chroma Cb block of the target block. In this case,R+1^(th) to N^(th) transform coefficients of the luma block and R+1^(th)to N^(th) transform coefficients of the chroma Cb block may be scanned.In the case that a non-zero transform coefficient is included in thescanned transform coefficients, the NSST index may be derived as 0. Inthe case that a non-zero transform coefficient is not included in thescanned transform coefficients, the bitstream may include a syntaxelement for the NSST index, and a value of the NSST index may be derivedby parsing the syntax element for the NSST index received through abitstream.

As another example, the NSST index may be used for inverse transform fortransform coefficients of a luma block of the target block and transformcoefficients of a chroma Cr block of the target block. In this case,R+1^(th) to N^(th) transform coefficients of the luma block and R+1^(th)to N^(th) transform coefficients of the chroma Cr block may be scanned.In the case that a non-zero transform coefficient is included in thescanned transform coefficients, the NSST index may be derived as 0. Inthe case that a non-zero transform coefficient is not included in thescanned transform coefficients, the bitstream may include a syntaxelement for the NSST index, and a value of the NSST index may be derivedby parsing the syntax element for the NSST index received through abitstream.

Meanwhile, based on a specific condition, a range of the NSST index maybe derived. For example, based on the specific condition, a maximumvalue of the NSST index may be derived, and the range may be derivedfrom 0 to the derived maximum value. The value of the derived NSST indexmay be included in the range.

For example, based on a size of the target block, a range of the NSSTindex may be derived. Particularly, a minimum width and a minimum heightmay be preconfigured. Based on the minimum width and a width of thetarget block, the minimum height and a height of the target block, arange of the NSST index may be derived. In addition, based on a specificvalue or a sample number of the target block, a range of the NSST indexmay be derived. The sample number may be a value of multiplying a widthand a height of the target block, and the specific value may bepreconfigured.

In addition, as another example, based on a type of the target block, arange of the NSST index may be derived. Particularly, a range of theNSST index may be derived based on whether the target block is anon-square block. In addition, a range of the NSST index may be derivedbased on a ratio between a width and a height of the target block and aspecific value. The ratio between a width and a height of the targetblock may be a value of a long side between the width and the height ofthe target block divided by a short side, and the specific value may bepreconfigured.

In addition, as another example, based on an intra prediction mode ofthe target block, a range of the NSST index may be derived.Particularly, a range of the NSST index may be derived based on whetherthe intra prediction mode of the target block is a non-directional intraprediction mode or a directional intra prediction mode. In addition,based on whether an intra prediction mode of the target block is anintra prediction mode included in category A or category B, a range of avalue of the NSST index may be determined. Here, the intra predictionmode included in category A and the intra prediction mode included incategory B may be preconfigured. As an example, the category A mayinclude No. 2 intra prediction mode, No. 10 intra prediction mode, No.18 intra prediction mode, No. 26 intra prediction mode, No. 34 intraprediction mode, No. 42 intra prediction mode, No. 50 intra predictionmode, No. 58 intra prediction mode and No. 66 intra prediction mode, andthe category B may include intra prediction modes except the intraprediction modes included in the category A.

In addition, as another example, based on information for a coretransform of the target block, a range of the NSST index may be derived.For example, a range of the NSST index may be derived based on an AMTflag that represents whether Adaptive Multiple core Transform (AMT) isapplied. In addition, a range of the NSST index may be derived based onan AMT horizontal flag indicating a horizontal transform kernel and anAMT vertical flag indicating a vertical transform kernel.

Meanwhile, in the case that a value of the NSST index is 0, the NSSTindex may indicate that NSST is not applied to the target block.

The decoding apparatus performs inverse transform for the transformcoefficients of the target block based on the NSST index and deriveresidual samples of the target block (step S1520).

For example, in the case that a value of the NSST index is 0, thedecoding apparatus may perform core transform for the transformcoefficients of the target block and derive the residual samples.

Particularly, the decoding apparatus may obtain an AMT flag thatrepresents whether Adaptive Multiple core Transform (AMT) is appliedfrom a bitstream.

In the case that a value of the AMT flag is 0, the decoding apparatusmay derive DCT type 2 as a transform kernel for the target block andderive the residual samples by performing inverse transform for thetransform coefficients based on DCT type 2.

In the case that a value of the AMT flag is 1, the decoding apparatusmay configure a transform subset for a horizontal transform kernel and atransform subset for a vertical transform kernel, derive a horizontaltransform kernel and a vertical transform kernel based on the transformindex information obtained from the bitstream and the transform subsets,and derive the residual samples by performing inverse transform for thetransform coefficients based on the horizontal transform kernel and thevertical transform kernel. Here, the transform subset for the horizontaltransform kernel and the transform subset for the vertical transformkernel may include DCT type 2, DCT type 7, DCT type 8 and/or DCT type 1as a candidate. In addition, transform index information may begenerated, and the transform index information may include an AMThorizontal flag indicating the horizontal transform kernel and an AMTvertical flag indicating the vertical transform kernel. Meanwhile, thetransform kernel may be called a transform type or a transform core.

In the case that a value of the NSST index is not 0, the decodingapparatus may derive a modified transform coefficients by performingNSST for the transform coefficients located in a top-left target area ofthe target block based on a reduced transform matrix indicated by theNSST index and derive the residual samples by performing core transformfor the target block including the modified transform coefficients. Asize of the reduced transform matrix may be R×N. The N may be a samplenumber of the top-left target area, and the R may be a reducedcoefficient. The R may be smaller than the N.

The core transform for the target block may be performed as below. Thedecoding apparatus may obtain an AMT flag that represents whetherAdaptive Multiple core Transform (AMT) is applied from a bitstream. Inthe case that a value of the AMT flag is 0, the decoding apparatus mayderive DCT type 2 as a transform kernel for the target block and derivethe residual samples by performing inverse transform for the transformcoefficients based on DCT type 2.

In the case that a value of the AMT flag is 1, the decoding apparatusmay configure a transform subset for a horizontal transform kernel and atransform subset for a vertical transform kernel, derive a horizontaltransform kernel and a vertical transform kernel based on the transformindex information obtained from the bitstream and the transform subsets,and derive the residual samples by performing inverse transform for thetarget block including the modified transform coefficients based on thehorizontal transform kernel and the vertical transform kernel. Here, thetransform subset for the horizontal transform kernel and the transformsubset for the vertical transform kernel may include DCT type 2, DCTtype 7, DCT type 8 and/or DCT type 1 as a candidate. In addition,transform index information may be generated, and the transform indexinformation may include an AMT horizontal flag indicating the horizontaltransform kernel and an AMT vertical flag indicating the verticaltransform kernel. Meanwhile, the transform kernel may be called atransform type or a transform core.

The decoding apparatus generates a reconstructed picture based on theresidual samples (step, S1530). The decoding apparatus may generate areconstructed picture based on the residual samples. For example, thedecoding apparatus may perform an inter prediction and an intraprediction for the target block based on the prediction informationreceived through a bitstream and derive prediction samples and generatethe reconstructed picture through a summation of the prediction samplesand the residual samples. Later, as occasion demands, in order toimprove subjective/objective image quality, the decoding apparatus mayapply the in-loop filtering process such as deblocking filtering and/orSAO process to the reconstructed picture as described above.

FIG. 16 schematically illustrates the decoding apparatus performing thevideo decoding method according to the present disclosure. The methodshown in FIG. 15 may be performed by the decoding apparatus shown inFIG. 16. Particularly, for example, the entropy decoder of the decodingapparatus shown in FIG. 16 may perform steps S1500 to S1510 of FIG. 15,the inverse transformer of the decoding apparatus shown in FIG. 16 mayperforms step S1520 of FIG. 15, and the adder of the decoding apparatusof FIG. 16 may perform step S1530 of FIG. 15. In addition, although itis not shown, the process of obtaining the prediction sample may beperformed by the predictor of the decoding apparatus.

According to the present disclosure described above, a range of NSSTindex may be derived based on a specific condition of a target block,and through this, bit amount for the NSST index may be reduced, andaccordingly, overall coding rate may be improved.

In addition, according to the present disclosure, a transmission ofsyntax element for NSST index may be determined based on transformcoefficients for a target block, and through this, bit amount for theNSST index may be reduced, and accordingly, overall coding rate may beimproved.

In the above-described embodiment, the methods are described based onthe flowchart having a series of steps or blocks, but the presentdisclosure is not limited to the order of the above steps or blocks.Some steps or blocks may occur simultaneously or in a different orderfrom other steps or blocks as described above. Further, those skilled inthe art will understand that the steps shown in the above flowchart arenot exclusive, that further steps may be included, or that one or moresteps in the flowchart may be deleted without affecting the scope of thepresent disclosure.

The method according to the present disclosure described above may beimplemented in software form, and the encoding apparatus and/or thedecoding apparatus according to the present disclosure may be includedin an apparatus that performs image processing such as a TV, a computer,a smartphone, a set-top box, a display device, and the like.

When the embodiments of the present disclosure is implemented withsoftware, the method described above may be implemented with a module(process, function, etc.) that performs the function described above.The module may be stored in a memory and executed by a processor. Thememory may be disposed to the processor internally or externally andconnected to the processor using a variety of well-known means. Theprocessor may include Application-Specific Integrated Circuits (ASICs),other chipsets, logic circuits, and/or data processors. The memory mayinclude Read-Only Memory (ROM), Random Access Memory (RAM), flashmemory, memory cards, storage media and/or other storage devices. Thatis, the embodiments described in the present disclosure may be performedby being implemented on a processor, a micro-processor, a controller ora chip. For example, the functional units shown in each drawing may beimplemented and performed on a computer, a processor, a micro-processor,a controller or a chip.

In addition, the decoding apparatus and the encoding apparatus to whichthe embodiments of the present disclosure are applied may be included ina multimedia broadcasting transmission/reception apparatus, a mobilecommunication terminal, a home cinema video apparatus, a digital cinemavideo apparatus, a surveillance camera, a video chatting apparatus, areal-time communication apparatus such as video communication, a mobilestreaming apparatus, a storage medium, a camcorder, a VoD serviceproviding apparatus, an Over the top (OTT) video apparatus, an Internetstreaming service providing apparatus, a three-dimensional (3D) videoapparatus, a teleconference video apparatus and a medical videoapparatus and may be used to process video signals and data signals. Forexample, the OTT video apparatus may include a game console, a blue-rayplayer, an internet access TV, a home theater system, a smartphone, atablet PC, a Digital Video Recorder (DVR), and the like.

Further, the processing method to which the present disclosure isapplied may be produced in the form of a computer-executed program andmay be stored in a computer-readable recording medium. The multimediadata having the data structure according to the present disclosure canalso be stored in a computer-readable recording medium. Thecomputer-readable recording medium includes all kinds of storage devicesand distributed storage devices in which computer-readable data isstored. The computer-readable recording medium may be, for example, aBlu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, anEPROM, an EEPROM, a RAM, a CD-Data storage devices. In addition, thecomputer-readable recording medium includes media implemented in theform of a carrier wave (for example, transmission over the Internet). Inaddition, the bit stream generated by the encoding method can be storedin a computer-readable recording medium or transmitted over a wired orwireless communication network. In addition, the embodiment of thepresent disclosure may be a computer program product by program codes,and the program codes may be performed by a computer by the embodimentof the present disclosure. The program codes may be stored on a carrierwhich is readable by a computer.

Further, a content streaming system to which the present disclosure isapplied may include an encoding server, a streaming server, a webserver, a media storage, a user device, and a multimedia input device.

The encoding server compresses content input from multimedia inputdevices such as smartphones, cameras, camcorders, etc. Into digital datato generate a bitstream and transmit the bitstream to the streamingserver. As another example, when multimedia input devices such assmartphones, cameras, camcorders, etc. directly generate a bitstream,the encoding server may be omitted. The bitstream may be generated by anencoding method or a bitstream generating method to which the presentdisclosure is applied, and the streaming server may temporarily storethe bitstream in the process of transmitting or receiving the bitstream.

The streaming server transmits multimedia data to the user device basedon a user request through the web server, and the web server serves asan intermediary for informing the user of what services are provided.When a user requests a desired service from the web server, the webserver delivers it to a streaming server, and the streaming servertransmits multimedia data to the user. Here, the content streamingsystem may include a separate control server, and in this case, thecontrol server controls a command/response between devices in thecontent streaming system.

The streaming server may receive content from a media repository and/oran encoding server. For example, when content is received from theencoding server, the content may be received in real time. In this case,in order to provide a smooth streaming service, the streaming server maystore the bitstream for a predetermined time.

Examples of the user device include a mobile phone, a smartphone, alaptop computer, a digital broadcasting terminal, a personal digitalassistant (PDA), a portable multimedia player (PMP), a navigationdevice, and a slate PC, a tablet PC, ultrabook, a wearable device (e.g.,smartwatch, glass glasses, head mounted display), a digital TV, adesktop computer, a digital signage, and so on. Each server in thecontent streaming system may operate as a distributed server, and inthis case, data received from each server may be processed in adistributed manner.

What is claimed is:
 1. A video decoding method performed by a decodingapparatus, comprising: obtaining residual information for a target blockfrom a bitstream; deriving transform coefficients of the target blockbased on the residual information; scanning (R+1)th to Nth transformcoefficients among the transform coefficients of the target block;deriving a non-separable transform index for the target block based onthe scanning of the (R+1)th to Nth transform coefficients; derivingresidual samples of the target block based on an inverse transform forthe transform coefficients of the target block, wherein the inversetransform is based on the non-separable transform index; generating areconstructed picture based on the residual samples; and applying adeblocking filtering on the reconstructed picture, wherein based on acase that a non-zero transform coefficient is included in the (R+1)th toNth transform coefficients, a value of the non-separable transform indexis derived as 0, wherein based on a case that the non-zero transformcoefficient is not included in the (R+1)th to Nth transformcoefficients, the inverse transform based on a non-separable transformis performed for coefficients in a top-left target area of the targetblock based on a transform matrix related to the non-separable transformindex, wherein a size of the transform matrix is R×N, wherein the N is anumber of samples in the top-left target area, and wherein the R issmaller than the N.
 2. The method of claim 1, wherein the step ofderiving the residual samples of the target block includes: based on avalue of the non-separable transform index being 0, performing coretransform for the transform coefficients of the target block andderiving the residual samples; and based on the value of thenon-separable transform index being not 0, deriving a modified transformcoefficients by performing the non-separable transform for the transformcoefficients located in the top-left target area of the target blockbased on the transform matrix related to the non-separable transformindex and deriving the residual samples by performing the core transformfor the target block including the modified transform coefficients. 3.The method of claim 1, wherein the step of deriving the non-separabletransform index for the target block includes: based on the non-zerotransform coefficient being not included in the (R+1)th to Nth transformcoefficients, deriving the value of the non-separable transform index byparsing a syntax element for the non-separable transform index includedin the bitstream.
 4. The method of claim 1, wherein a range of thenon-separable transform index is derived based on a minimum width and awidth of the target block, a minimum height and a height of the targetblock, and wherein the minimum width and the minimum height arepreconfigured.
 5. The method of claim 1, wherein a range of thenon-separable transform index is derived based on a specific value and anumber of samples in the target block, and wherein the number of samplesis a value of multiplying a width and a height of the target block, andthe specific value is preconfigured.
 6. The method of claim 1, wherein arange of the non-separable transform index is derived based on whetherthe target block is a non-square block.
 7. The method of claim 1, basedon the non-separable transform index being used for the inversetransform for transform coefficients of a luma block of the targetblock, transform coefficients of a chroma Cb block and transformcoefficients of a chroma Cr block, R+1th to Nth transform coefficientsof the luma block, R+1th to Nth transform coefficients of the chroma Cbblock and R+1th to Nth transform coefficients of the chroma Cr block arescanned, and based on a non-zero transform coefficient being included inthe scanned transform coefficients, the value of the non-separabletransform index is derived as
 0. 8. The method of claim 1, wherein annon-separable transform index indicator for a target CTU including thetarget block is generated from the bitstream, and wherein thenon-separable transform index indicator is related to whethernon-separable transform is enabled in the target CTU.
 9. A videoencoding method performed by an encoding apparatus, comprising: derivingresidual samples of a target block; deriving transform coefficients ofthe target block by performing a transform based on the residualsamples; scanning (R+1)th to Nth transform coefficients among thetransform coefficients of the target block; determining whether toencode a non-separable transform index for the transform coefficientsbased on the scanning of the (R+1)th to Nth transform coefficients; andencoding residual information including information related to thetransform coefficients and information related to the non-separabletransform index, wherein based on a case that a non-zero transformcoefficient is included in the (R+1)th to Nth transform coefficients,the non-separable transform index is determined not to encode, whereinbased on a case that the non-zero transform coefficient is not includedin the (R+1)th to Nth transform coefficients, the non-separabletransform index is determined to encode, wherein based on a value of thenon-separable transform index not being equal to 0, the transform basedon a non-separable transform is performed for coefficients in a top-lefttarget area of the target block based on a transform matrix related tothe non-separable transform index, wherein a size of the transformmatrix is R×N, wherein the N is a number of samples in the top-lefttarget area, and wherein the R is smaller than the N.
 10. The method ofclaim 9, wherein the step of deriving the transform coefficients of thetarget block by performing the transform for the residual samplesincludes: determining whether to apply the non-separable transform forthe target block; based on the non-separable transform being applied tothe target block, deriving modified transform coefficients by performingcore transform for the residual samples and deriving the transformcoefficients of the target block by performing the non-separabletransform for the modified transform coefficients located in a top-lefttarget area of the target block based on a transform matrix; and basedon the non-separable transform being not applied to the target block,derive the transform coefficients of the target block by performing thecore transform for the residual samples.
 11. The method of claim 9,wherein a range of the non-separable transform index is derived based ona minimum width and a width of the target block, a minimum height and aheight of the target block, and wherein the minimum width and theminimum height are preconfigured.
 12. The method of claim 9, wherein arange of the non-separable transform index is derived based on aspecific value and a number of samples in the target block, and whereinthe number of samples is a value of multiplying a width and a height ofthe target block, and the specific value is preconfigured.
 13. Anon-transitory computer-readable digital storage medium storing abitstream generated by a video encoding method, the method comprising:deriving residual samples of a target block; deriving transformcoefficients of the target block by performing a transform based on theresidual samples; scanning (R+1)th to Nth transform coefficients amongthe transform coefficients of the target block; determining whether toencode a non-separable transform index for the transform coefficientsbased on the scanning of the (R+1)th to Nth transform coefficients; andencoding residual information including information related to thetransform coefficients and information related to the non-separabletransform index to output the bitstream, wherein based on a case that anon-zero transform coefficient is included in the (R+1)th to Nthtransform coefficients, the non-separable transform index is determinednot to encode, wherein based on a case that the non-zero transformcoefficient is not included in the (R+1)th to Nth transformcoefficients, the non-separable transform index is determined to encode,wherein based on a value of the non-separable transform index not beingequal to 0, the transform based on a non-separable transform isperformed for coefficients in a top-left target area of the target blockbased on a transform matrix related to the non-separable transformindex, wherein a size of the transform matrix is R×N, wherein the N is anumber of samples in the top-left target area, and wherein the R issmaller than the N.
 14. A transmission method of data for a video, themethod comprising: obtaining a bitstream for the video, wherein thebitstream is generated based on deriving residual samples of a targetblock, deriving transform coefficients of the target block by performinga transform based on the residual samples, scanning (R+1)th to Nthtransform coefficients among the transform coefficients of the targetblock, determining whether to encode a non-separable transform index forthe transform coefficients based on the scanning of the (R+1)th to Nthtransform coefficients, and encoding residual information includinginformation related to the transform coefficients and informationrelated to the non-separable transform index; and transmitting the datacomprising the bitstream, wherein based on a case that a non-zerotransform coefficient is included in the (R+1)th to Nth transformcoefficients, the non-separable transform index is determined not toencode, wherein based on a case that the non-zero transform coefficientis not included in the (R+1)th to Nth transform coefficients, thenon-separable transform index is determined to encode, wherein based ona value of the non-separable transform index not being equal to 0, thetransform based on a non-separable transform is performed forcoefficients in a top-left target area of the target block based on atransform matrix related to the non-separable transform index, wherein asize of the transform matrix is R×N, wherein the N is a number ofsamples in the top-left target area, and wherein the R is smaller thanthe N.